Remove references to Steam Community XML feeds

[dpi]

This is probably more of a tracking issue since its not possible to get all data that is exposed in the XML feeds in the WebAPI.

The rate limiting on Steam Community is getting progressively more strict. Overcoming this rate limiting would involve moving to the WebAPI as the only source of data. I would like to think that this is on the horizon for Condenser. What are the chances of starting to implement WebAPI only, and removing data that is no longer exposed?

This thread on the Steam forums [2] seems to indicate some word of discontinuation of the XML feeds. Although not official, its not beyond belief.

they are now rate limited. From what I read, Valve plans on deprecating these feeds and the new rate limit is there to reduce the load on Steam Community.

This move is going to require major version release (v2). I am interested in getting the ball rolling. At least with development of the PHP library.

[Yeggstry]

This is already in progress as a result of the additions I am making, but I agree we need to move to the WebAPI as its much more stable.

Problem is that it doesn't cover some fundamental data required. For instance, you cannot get a list of games that a user owns from the WebAPI. You'll find that all of the steam stats websites are struggling with this issue and that there has been a lot of requests to provide this in the WebAPI.

It would be great to have a hand on the PHP side of ISteamUserStats if you're interested. I've only dabbled in PHP in my spare time so I'm not an expert :)

[koraktor]

I hope that the Web API will get some improvements over the coming weeks and months so this can make it into 2.0.0. XML access won't be removed if there's no equivalent data source. The quality of service has been degrading constantly over the last months resulting in more and more HTTP 503 errors which is kind of a show stopper.

[Yeggstry]

Is the improvement of the Web API something that Valve are working on? Insight into that would be useful but I can't find any central place for the discussion of the Web API that is up-to-date.

[koraktor]

I really hope so. I did already trigger some Valve guys to at least open up a issue tracker for the Web API about a week ago. As far as I know they still have to decide if this is happening, but @alfred-valve indicated that this is in the responsibility of each team.

On the other hand, I don't know if there's really one Web API team. It seems like e.g. the interfaces for DotA 2 are updated together with the game's updates. So the Web API might be developed by the products' teams as well.

[Yeggstry]

Since the XML feeds have now been deprecated, its worth mapping the feed data with the equivalent WebAPI calls:

Player profile

• main profile details - most of the information can be found from GetPlayerSummaries operation in ISteamUser, with the ban information found in GetPlayerBans in ISteamUser. I can't see where to get the steam rating or number of hours played in the last 2 weeks, but I think these could be worked out from other data. The summary doesn't seem to be available at all.
• mostPlayedGames (which is actually the most recently played games) seems to be covered by the GetRecentlyPlayedGames operation from IPlayerService.
• groups - a list of group IDs for a user can be found in GetUserGroupList in ISteamUser. The primary group can be found in the GetPlayerSummaries in ISteamUser.

Games List

• GetOwnedGames from IPlayerService provides the appId, total time played and time played in the last 2 weeks, which is the user-specific information we need from the old games list feed.
• include_appinfo flag (0 or 1) provides the name and logos. Perhaps this should also be provided in another service or included in ISteamApps -> GetAppList, since I can't see any other way of getting basic app details.

[TheSeg]

As I start working on making the PHP offering streamlined, I'm trying to figure out how much dependency we want with the deprecated XML fields that have more accurate fields. For example, "onlineState" is replaced with a combination of "personastate" & "gameid". Do we want to have more functions to retrofit the old data, or just drop the old data and update the user functions like $user->isInGame() to use the new data.

[koraktor]

The main goal when switching from XML to Web API is probably to not drop existing features. If there's a better alternative for some of them, don't hesitate to use it.

[Yeggstry]

Should the offering between the different languages be consistent? Granted most of what I've done already is adding the WebAPI operations in, in preperation for this particular task.

I appreciate that the implementation between Java and Ruby isn't consistent at the moment, but if the main class API is the same does it matter?

[TheSeg]

Because I needed to do this anyway, I'm making a chart that defines the differences between XML and WebAPI data. Once I"m done I'll share results.

I think from there we have to decided how much we keep of the old and bring in the new. There's some data gone from WebAPI and some data that's changed to machine consumable formats. Once I get it done, we can talk more about it.

[TheSeg]

Steam WebAPI vs. XML

Here's the Google Doc of the compare between WebAPI and XML. At this writing I got everything on the user Profile XML mapped to WebAPI. Here's my overview of findings:

Almost all fields exist

Almost all the fields exist in WebAPI. Most are 1 to 1 relationships, though some though different WebAPI calls. The spreadsheet highlights them.

Some fields are adjusted

Some have different datapoints which can arrive to the same conclusions.

• location -> locCountryCode, locStateCode, locCityId

  The user's location in XML is a string, but in WebAPI is three fields of identifiers.

• privacyState -> communityVisibilityState

  This field is replaced with WebAPI's communityVisibilityState, which offers more details as an int.

• Gameplay moved to IPlayerService::GetRecentlyPlayedGames & ``IPlayerService::GetOwnedGames`

  Between these two WebAPI functions, we get all the recent gameplay data from XML, plus owned games and per-game time play.

Fields & Data with no WebAPI

• Group Details

  While WebAPI provides the IDs of what groups & the primary group of the user's profile, the details of the group aren't exposed on WebAPI yet. So we can point to groups and get the info, but from XML.

• Total Hours played in last 2 weeks.

  We can get the 2 week data from an individual game, but there's no combined time for 2 weeks. There is however lifetime play time across all games.

• Limited Account Status

  Couldn't find a way to get the limited account status. All other bans are visible though. Perhaps I'm missing a logic deduction?

[koraktor]

@Yeggstry Consistent internals of the different implementations aren't a must-have (sometimes not even possible), but as I have to maintain three different code bases I'd love to have homogeneous code for all. Even if there might be 3 pull requests from 3 different developers that implement the same feature in 3 different ways I'd probably try to find a common "base implementation" in the long-term.

@TheSeg Wow, thanks. Seems like a valuable source for every one of use. Although I already know these it's good to see it written down. We should really try to keep this up-to-date if something changes. Would you mind giving write-acces to me (and maybe others interested in this concern)?

Altogether, I really want to get this done, but in the moment my time to work on Steam Condenser is somewhat limited.

[TheSeg]

@koraktor No problem! It's keeping me current on Steam while I'm between employment. Totally can give you write access! I need a Gmail address. If you just give the username, I can assume the @gmail.com part.

[koraktor]

I think you can also guess the user name, or just look into the Git logs. 😏

[TheSeg]

@koraktor Done!

[TheSeg]

Sorry for the delay, but I created something that may be of interest to everyone:

Steam WebAPI Proposal

I needed a public demonstration of my API development skills, so I created this site for proposing how hubs and game app data could work for WebAPI. Have a look though and feel free to comment!

[CrimsonTautology]

In reference to #268, I went and tried the XML api again just now and it appears to be working again.