Author Topic: Live Rankings/Scoring (Research)  (Read 1541 times)

0 Members and 1 Guest are viewing this topic.

Offline masterclass

  • Tennis Pro
  • *****
  • Posts: 366
Live Rankings/Scoring (Research)
« on: May 13, 2013, 04:26:38 AM »
Live Rankings Research thread.
Started to investigate various ways to retrieve live rankings better in an automated fashion.
For example, to get Futures results.

Marv, you are welcome to please explain your needs and requirements further.

I saw the ITF page from where you are trying to retrieve results and it is not straight forward due to the results being gathered and displayed in a flash application. The HTML page calls the flash immediately, and the page is all flash :)

We will have to explore the ways and means to try and get at the data.

Respectfully,
masterclass
« Last Edit: May 13, 2013, 05:46:29 AM by masterclass »
Legends of Tennis

Offline Slasher1985

  • Global Moderator
  • Tennis God
  • ******
  • Posts: 572
  • Gender: Male
  • Rankings Wizard
Re: Live Rankings (Research)
« Reply #1 on: May 13, 2013, 04:56:59 AM »
Well, here is the solution, sir John. Although, it was not easy. The page loads the following feed:

http://scores.itftennis.com/feed.php?type=days&from=1368392400000&to=1368478800000&key=a017c9d5e6d127b073e6db56fb5f0be7&language=en&config=9651&bookid=8640&phrasesid=&clientid=597&mc=3&c=2

Using that feed, I was able to identify the tournaments and the matches being played. The from= and to= from the address are the UNIX timestamps you want to load matches between. For instance, I was able to identify the following matches from the Argentine futures tournament:

Code: [Select]
<t i="25862" u="0" n="ARGENTINA F6 FUTURES" c="ar" a3="ARG" a3o="ARG" s="100000" g="1" gc="1" b="3" ei="1951" eg="0" t="1" pm="10000$">
<m b="3849241"    lu="0" mc="0" cs="0" pp="1" iv="1"><d>1368457200</d>
<t1 i="5438199" u="57289" r="484" c="BOL" a2="bo"  sc="0" sx="0">Dellien, Hugo</t1>
<t2 i="5419669" u="42691" r="787" c="ARG" a2="ar"  sc="0" sx="0">Ortiz, Juan Pablo</t2>
<s c="0">Not started</s><w>0</w>
</m>
<m b="3849243"    lu="0" mc="0" cs="0" pp="1" iv="1"><d>1368457200</d>
<t1 i="5719844" u="54467" r="631" c="ARG" a2="ar"  sc="0" sx="0">Perez Mota, Mauricio</t1>
<t2 i="5911965" u="66086" r="740" c="ARG" a2="ar"  sc="0" sx="0">Kicker, Nicolas</t2>
<s c="0">Not started</s><w>0</w>
</m>
<m b="3849237"    lu="0" mc="0" cs="0" pp="1" iv="1"><d>1368457200</d>
<t1 i="1452604" u="32526" r="399" c="CHI" a2="cl"  sc="0" sx="0">Saavedra-Corvalan, Cristobal</t1>
<t2 i="5719966" u="54573" r="626" c="ARG" a2="ar"  sc="0" sx="0">Coria, Federico</t2>
<s c="0">Not started</s><w>0</w>
</m>
<m b="3849245"    lu="0" mc="0" cs="0" pp="1" iv="1"><d>1368457200</d>
<t1 i="5264095" u="38182" r="776" c="ECU" a2="ec"  sc="0" sx="0">Vivanco, Juan-Sebastian</t1>
<t2 i="655617" u="16740" r="366" c="ARG" a2="ar"  sc="0" sx="0">Amado, Juan-Pablo</t2>
<s c="0">Not started</s><w>0</w>
</m>
<m b="3849239"    lu="0" mc="0" cs="0" pp="1" iv="1"><d>1368462600</d>
<t1 i="5891993" u="65606" r="1126" c="ARG" a2="ar"  sc="0" sx="0">Agamenone, Franco</t1>
<t2 i="5720987" u="52959" r="629" c="JPN" a2="jp"  sc="0" sx="0">Makiguchi, Ryusei</t2>
<s c="0">Not started</s><w>0</w>
</m>
</t>
<t i="25863" u="0" n="ARGENTINA F6 FUTURES" c="ar" a3="ARG" a3o="ARG" s="100000" g="1" gc="1" b="3" ei="1951" eg="0" t="2" pm="10000$">
<m b="3849293"    lu="0" mc="0" cs="0" r="2" pp="1" iv="1"><d>1368462600</d>
<t1 i="6293692" u="75819"  sc="0" sx="0">Cuevas M / Londero J I</t1>
<t2 i="6647586" u="98729"  sc="0" sx="0">Bas A E / Pellegrini J M</t2>
<dt>
<t k="h1" i="5269493" c="URU" a2="uy" sc="0" sx="0" />
<t k="h2" i="5630207" c="ARG" a2="ar" sc="0" sx="0" />
<t k="a2" i="6646642" c="ARG" a2="ar" sc="0" sx="0" />
<t k="a1" i="6647699" c="ARG" a2="ar" sc="0" sx="0" />
</dt>
<s c="0">Not started</s><w>0</w>
</m>
</t>

I have enough data here for now to start creating an automated procedure that will gather the data needed to live update Futures results into the rankings. Also, this is the key to introduce live scoring to Futures matches into T4U, so I guess Scott may want to see this in detail. :king:

EDIT: That link is metadata from xml, not a specific website, so your browser will either not show anything or show an error. You must see the source code to gather the data.
« Last Edit: May 13, 2013, 04:59:37 AM by Slasher1985 »

Offline masterclass

  • Tennis Pro
  • *****
  • Posts: 366
Re: Live Rankings (Research)
« Reply #2 on: May 13, 2013, 05:04:05 AM »
Very good start Sir Marv!  Now I assume you just want completed matches, and not ones that have not started or are in progress, correct?

Respectfully,
masterclass
Legends of Tennis

Offline Slasher1985

  • Global Moderator
  • Tennis God
  • ******
  • Posts: 572
  • Gender: Male
  • Rankings Wizard
Re: Live Rankings (Research)
« Reply #3 on: May 13, 2013, 05:08:43 AM »
Very good start Sir Marv!  Now I assume you just want completed matches, and not ones that have not started or are in progress, correct?

Respectfully,
masterclass

Yes indeed, sir John. So those would be the ones having this tag <s c="100">Ended</s> in their midst. Then the score:

<s t="period1"><t1>6</t1><t2>4</t2>
<s t="period2"><t1>2</t1><t2>6</t2>
<s t="period3"><t1>5</t1><t2>7</t2>

Offline masterclass

  • Tennis Pro
  • *****
  • Posts: 366
Re: Live Rankings (Research)
« Reply #4 on: May 13, 2013, 05:20:04 AM »
Ideally, one would be able to pass in the proper parameters into the query string to restrict the data returned from the server to what you want.
But if that's not possible, then another way to do that would be using xpath to filter the XML data as it is returned.
Do you know how to do that?

Respectfully,
masterclass
« Last Edit: May 13, 2013, 05:21:56 AM by masterclass »
Legends of Tennis

Offline Slasher1985

  • Global Moderator
  • Tennis God
  • ******
  • Posts: 572
  • Gender: Male
  • Rankings Wizard
Re: Live Rankings (Research)
« Reply #5 on: May 13, 2013, 05:35:22 AM »
Ideally, one would be able to pass in the proper parameters into the query string to restrict the data returned from the server to what you want.
But if that's not possible, then another way to do that would be using xpath to filter the XML data as it is returned.
Do you know how to do that?

Respectfully,
masterclass

Well, I'm going to do this in the same manner as I do with the live scoring of ATP and Challenger events.

First, I restrict the portion of the file I need (the tournament currently being updated).
I search for completed matches played in singles.
I determine the winning player, losing player and the score.
All this is done by "scratching" the XML file (I notice the key tags and extract the data from them). VBA from Excel does not really have high level functions like xpath, but I can do this the hard way just as well.

Now will come the part on which I will use the already constructed draw to determine the round (because the ITF does not provide the round in this live stream).
After that, I write the match result into a common file (also used by the ATP and Challenger events).
When the specific Futures tournament is updated, it looks into the common file, notes the result and updates the excel file.

This is notably simple on my part. I think it would be a little harder for Scott to make this in a viable live scoring feature for the main site. I'm certain he'd be interested in this, if he notices this thread.
« Last Edit: May 13, 2013, 05:36:27 AM by Slasher1985 »

Offline Slasher1985

  • Global Moderator
  • Tennis God
  • ******
  • Posts: 572
  • Gender: Male
  • Rankings Wizard
Re: Live Rankings/Scoring (Research)
« Reply #6 on: May 13, 2013, 05:53:15 AM »

Offline masterclass

  • Tennis Pro
  • *****
  • Posts: 366
Re: Live Rankings/Scoring (Research)
« Reply #7 on: May 13, 2013, 06:00:35 AM »
How about this.

I've managed to repair the XML tree and now I can see it without error:

http://scores.itftennis.com/feed.php?type=days&from=1368366681000&to=1368453081000&key=a017c9d5e6d127b073e6db56fb5f0be7&language=en&config=9651&bookid=8640&mc=3


:) Yep, I had already seen that by removing the 'c=2' the xml was properly formatted.  Sorry, I should have mentioned that to you. But I see you got it anyway...

Respectfully,
masterclass
Legends of Tennis

Offline masterclass

  • Tennis Pro
  • *****
  • Posts: 366
Re: Live Rankings (Research)
« Reply #8 on: May 13, 2013, 06:25:49 AM »
Ideally, one would be able to pass in the proper parameters into the query string to restrict the data returned from the server to what you want.
But if that's not possible, then another way to do that would be using xpath to filter the XML data as it is returned.
Do you know how to do that?

Respectfully,
masterclass

Well, I'm going to do this in the same manner as I do with the live scoring of ATP and Challenger events.

First, I restrict the portion of the file I need (the tournament currently being updated).
I search for completed matches played in singles.
I determine the winning player, losing player and the score.
All this is done by "scratching" the XML file (I notice the key tags and extract the data from them). VBA from Excel does not really have high level functions like xpath, but I can do this the hard way just as well.

Now will come the part on which I will use the already constructed draw to determine the round (because the ITF does not provide the round in this live stream).
After that, I write the match result into a common file (also used by the ATP and Challenger events).
When the specific Futures tournament is updated, it looks into the common file, notes the result and updates the excel file.

This is notably simple on my part. I think it would be a little harder for Scott to make this in a viable live scoring feature for the main site. I'm certain he'd be interested in this, if he notices this thread.

Ok, I know there is a  direct FILTERXML(xml, xpath) in Excel, but it looks like you are using VBA to extract the data, manipulate it, combining with draw, and then writing to a common file, and then use that file to update data in an Excel file.

So I think VBA also has xpath functions to deal with XML that might make your job a bit easier.
Using something like:
Code: [Select]
' Some dim statements'

Dim xml As MSXML2.DOMDocument60
Dim nodes As MSXML2.IXMLDOMNodeList
Dim node As MSXML2.IXMLDOMNode

'Get the xml from the url'
'set xml from the response code'

'Get the specific xml you need'

Set nodes = xml.SelectNodes(xpath expression)

'Loop through the nodes returned and extract specific data'

For Each node In nodes   
        Debug.Print node.SelectSingleNode(nodeElementname).Text
       ' or use  xmlText(node.SelectSingleNode("nodeElementName")) and set it to a value
Next node

Of course the actual code would be more than that to extract all the data you need

Respectfully,
masterclass
« Last Edit: May 13, 2013, 07:20:58 AM by masterclass »
Legends of Tennis

Offline Slasher1985

  • Global Moderator
  • Tennis God
  • ******
  • Posts: 572
  • Gender: Male
  • Rankings Wizard
Re: Live Rankings/Scoring (Research)
« Reply #9 on: May 13, 2013, 07:39:07 AM »
This has come really useful, General. Yet I have reached another complication. The public key the site uses changes from time to time. I have a problem automatically identifying the new key now.

Offline masterclass

  • Tennis Pro
  • *****
  • Posts: 366
Re: Live Rankings/Scoring (Research)
« Reply #10 on: May 13, 2013, 09:06:04 AM »
Ugh.. that's a problem. 
It looks like the key is generated based on the alias param when you access the web site url
   
   http://scores.itftennis.com/?alias=itf

Then in the console, one can see the feed url with the generated key.   

It would be nice if there were a known api to use to get the key... :)

I tried passing in the alias param into the feed url, and that doesn't work.

It might have to be a two step process, access the site, get the correct feed url with the correct key.  But then that has to be updated regularly.  Key probably lasts a set period, maybe an hour or so.

Respectfully,
masterclass
Legends of Tennis

Offline Slasher1985

  • Global Moderator
  • Tennis God
  • ******
  • Posts: 572
  • Gender: Male
  • Rankings Wizard
Re: Live Rankings/Scoring (Research)
« Reply #11 on: May 13, 2013, 09:13:21 AM »
Ugh.. that's a problem. 
It looks like the key is generated based on the alias param when you access the web site url
   
   http://scores.itftennis.com/?alias=itf

Then in the console, one can see the feed url with the generated key.   

It would be nice if there were a known api to use to get the key... :)

I tried passing in the alias param into the feed url, and that doesn't work.

It might have to be a two step process, access the site, get the correct feed url with the correct key.  But then that has to be updated regularly.  Key probably lasts a set period, maybe an hour or so.

Respectfully,
masterclass


Wait, I'm not seeing the key using that link. It just loads the main site once more. Where should I look?

Offline masterclass

  • Tennis Pro
  • *****
  • Posts: 366
Re: Live Rankings/Scoring (Research)
« Reply #12 on: May 13, 2013, 09:45:42 AM »
Ugh.. that's a problem. 
It looks like the key is generated based on the alias param when you access the web site url
   
   http://scores.itftennis.com/?alias=itf

Then in the console, one can see the feed url with the generated key.   

It would be nice if there were a known api to use to get the key... :)

I tried passing in the alias param into the feed url, and that doesn't work.

It might have to be a two step process, access the site, get the correct feed url with the correct key.  But then that has to be updated regularly.  Key probably lasts a set period, maybe an hour or so.

Respectfully,
masterclass


Wait, I'm not seeing the key using that link. It just loads the main site once more. Where should I look?


Which browser are you using Marv?  I'm using Firefox.
After I use the link, I click on Tools > Web Developer and toggle on the Web Console.
The key is actually used prior to the feed call.  I was trying to see if one of the early calls generated the key, but I'm not sure which it is.
My guess is that it is somewhere in the flash code.

Respectfully,
masterclass
Legends of Tennis

Offline masterclass

  • Tennis Pro
  • *****
  • Posts: 366
Re: Live Rankings/Scoring (Research)
« Reply #13 on: May 14, 2013, 06:15:59 AM »
Marv, maybe it's time to consider another site? 

This one looks promising:  http://www.tennisexplorer.com/results/

Looks like the live score data provider for the above site might be:  http://Livescore.in
Seems like they offer a free service as well...

Respectfully,
masterclass
« Last Edit: May 14, 2013, 06:50:25 AM by masterclass »
Legends of Tennis

Offline Slasher1985

  • Global Moderator
  • Tennis God
  • ******
  • Posts: 572
  • Gender: Male
  • Rankings Wizard
Re: Live Rankings/Scoring (Research)
« Reply #14 on: May 15, 2013, 09:11:45 AM »
Sorry, I was away. Gonna continue with this tomorrow. I have not given up on the initial solution and will not try other sites yet. That solution is more permanent than livescore sites on the web. :)

Offline Slasher1985

  • Global Moderator
  • Tennis God
  • ******
  • Posts: 572
  • Gender: Male
  • Rankings Wizard
Re: Live Rankings/Scoring (Research)
« Reply #15 on: May 16, 2013, 02:34:10 AM »
Thanks a lot John. You've been tremendously helpful. I have figured out how to get the key, by using click events on the website using the Firefox console and copying the text on the clipboard, extracting the key from within. Quite unorthodox but extremely efficient I might add.

Now, the rest will be quite easy, following your earlier advice, and this should be done this week. :)

Offline Slasher1985

  • Global Moderator
  • Tennis God
  • ******
  • Posts: 572
  • Gender: Male
  • Rankings Wizard
Re: Live Rankings/Scoring (Research)
« Reply #16 on: May 16, 2013, 07:25:05 AM »
The prototype update will start in 30 minutes, and this will be testing for the rest of this week. Implementation is ready. :)~

Offline masterclass

  • Tennis Pro
  • *****
  • Posts: 366
Re: Live Rankings/Scoring (Research)
« Reply #17 on: May 17, 2013, 05:52:22 AM »
Thanks a lot John. You've been tremendously helpful. I have figured out how to get the key, by using click events on the website using the Firefox console and copying the text on the clipboard, extracting the key from within. Quite unorthodox but extremely efficient I might add.

Now, the rest will be quite easy, following your earlier advice, and this should be done this week. :)

Well done Marv.  Persistence is the mother of invention.  :worthy:
How often are you refreshing the key?

Respectfully,
masterclass
Legends of Tennis

Offline Slasher1985

  • Global Moderator
  • Tennis God
  • ******
  • Posts: 572
  • Gender: Male
  • Rankings Wizard
Re: Live Rankings/Scoring (Research)
« Reply #18 on: May 17, 2013, 11:27:12 AM »
How often are you refreshing the key?

Respectfully,
masterclass

Well, if the variable memorizing the key is null, I refresh it, and if the feed returns Authentication failed, the key is refreshed. That's about it. Minimum refreshing possible.

Offline Slasher1985

  • Global Moderator
  • Tennis God
  • ******
  • Posts: 572
  • Gender: Male
  • Rankings Wizard
Re: Live Rankings/Scoring (Research)
« Reply #19 on: May 20, 2013, 02:38:15 AM »
I noticed a new rule that exists only in the Race rankings. The number of countable non-mandatory tournaments is not 18 - mandatories played, but rather, 6 + mandatories hosted YTD - mandatories played. I see that live-tennis.eu didn't implement this rule, so it will only exist in the case of my rankings.

Notice Paire. I have him at 975 and live-tennis.eu the same. But the official is 955, because he is not allowed to count more than 6 non-mandatories, since he played all 5 possible mandatories. So, his 20 from Estoril does not count. The same with Rosol, his Auckland 6 points do not count.

I am making the modifications now, and you should be seeing this in.