Discussion Has any one experimented with OCR tools for scraping the commodities market prices?

FD: "These tools messes up our market algorithms so the game is not enjoyable. Thus, we are forbidding them, but may think of an API that can be used to produce output that doesn't completely mess up the algorithms."

Modders: "Screw you, I do what I want, and I don't care if it destroys the game for lots of people".


... seriously? Stop arguing, because the argument you present is a stupid argument. The game systems are built to emulate a real world, and you are taking measures to make it unreal. For what? Because you are too lazy to play the game the way it is meant to be played? Because you enjoy destroying the work of others? Because you like to detract from the joy others get who want to get "the real" experience? How can you possibly justify continuing, when you know FD are against it, and you know the reasons why?


you seriously believe real world is the way to go ok....in the real world you use every legal method to give YOURSELF the most information before money changes hands......you dont guess ..... you dont waist time on things you cant get accurate fast information on ....you think wallstreet traders dont have apps?....you think bankers arent using every technology they can to wring ever dime they can ?

seriously get a clue.

im fine with them wantin to do it inhouse so that it doesnt effect the gamecode....... but a tool is needed and to ban the others before they have theres ready is....shortsighted at best
 
you seriously believe real world is the way to go ok....in the real world you use every legal method to give YOURSELF the most information before money changes hands......you dont guess ..... you dont waist time on things you cant get accurate fast information on ....you think wallstreet traders dont have apps?....you think bankers arent using every technology they can to wring ever dime they can ?

seriously get a clue.

im fine with them wantin to do it inhouse so that it doesnt effect the gamecode....... but a tool is needed and to ban the others before they have theres ready is....shortsighted at best

I agree with your sentiment here. It's so weirdly naive to me, that the game implements a financial system and then people complain that there's ways found to, without actually cheating*, get better results out of it faster.

Especially since we can see the number of threads on "I give up" and "it's too hard". Put a carrot in front of a starving donkey, it doesn't matter how rotten the carrot may be, it's going to get eaten.

* Regardless of the memory "hack" required, it's read-only. Cheating to me, would be using a memory hack to inject credits, or give the player a ship they didn't buy, or auto-aiming to shoot the target every single time etc.. etc.. or jump without using fuel. Whereas the trading tool is simply time-saving on iterative work, rather than skipping entertaining gameplay.

I guess it comes down to whether you're a gambler at heart and want to use some form of intuitive gut instinct to feel good about a successful shot in the dark, vs studying it to take the risk out of things, and these tools take the risk out, so the gambler psychology feels this is "unfair", perhaps?

Even further along, this is the "flaw" in a way, of having everything be purchaseable, rather than better gear through loot drops. With such a heavy emphasis on "credits buys everything/anything", and given credit requirements don't currently scale, then the means to get credits faster impacts everything across the game. You don't need to be a particularly good pirate, just buy the best gear, so trade to get the cash first.
 
Last edited:

wolverine2710

Tutorial & Guide Writer
To make it nice and obvious that it's reasonably trivial:

https://imgur.com/a/YUNzQ

That's undistorted text using a very simple OpenCV python script (10 lines) which just generates a perspective transform and applies it, then thresholds it. OCR software likes the resulting image a lot. It'd be trivial to slice it up using the nice clear lines and process it. I also played around a bit with taking a screenshot in the station and another in the hangar; this gives a very good image as you can combine them and find the common pixels, which will always be the HUD and bits of ship. You could run this script in near-realtime using a screen grabber and tie it in with some keyboard controls to automatically scroll through the list until it's gotten everything.

I'm showing this because it was 30 or so minute's work to get that far and I'm not even a computer vision specialist; anyone who wants to get data out of the game is going to do it. Encryption won't stop them. Screaming for these tools to be banned and blocked is just going to drive this stuff underground.

With regard to common pixels. I just googled 2 minutes and found something relevant, though it does the reverse. See here.It seems powerful tools like imagemagick (also for windows) and others could be really useful if/when we go down the OCR route.
 
Stop arguing, because the argument you present is a stupid argument. The game systems are built to emulate a real world, and you are taking measures to make it unreal. For what?
To retain ED being a fun game. A game is about being fun and interesting to play. That includes curbing some toxic behaviours from the "real world". Those tools are akin to High Speed Trading in real life stocks, it changes the market and forces you to do likewise to stay competitive.
These tools also dispense you to do any meaningful work and research ingame and remove the essential part of trading which is searching markets and making the decision of what to trade where. This is essentially botting the research process and removes all player skill frm the equation (skill not only being able to click very fast on the keyboard, but also using your brain)
you are welcome to keep track of data with Excel and sharing the files, or manually update a web site, at least it requires some action and thingking on your side.
The thing is, using these tools is just being lazy and being rewarded for it. That's not how games work.
 
Well, IMO there is no point in using existing OCR, since it's a known consistent font and thus can very quickly be done (just by using shape matching) based on said font (yes I'm simplifying the algorithm vastly)

I may if I get chance over the weekend have a quick play to see what can be thrown together, but I expect it will be quite simple to interpret the data.
 

wolverine2710

Tutorial & Guide Writer
Well, IMO there is no point in using existing OCR, since it's a known consistent font and thus can very quickly be done (just by using shape matching) based on said font (yes I'm simplifying the algorithm vastly)

I may if I get chance over the weekend have a quick play to see what can be thrown together, but I expect it will be quite simple to interpret the data.

Looking forward to your results!!!
 
These tools also dispense you to do any meaningful work and research ingame and remove the essential part of trading which is searching markets and making the decision of what to trade where. This is essentially botting the research process and removes all player skill frm the equation (skill not only being able to click very fast on the keyboard, but also using your brain)
you are welcome to keep track of data with Excel and sharing the files, or manually update a web site, at least it requires some action and thingking on your side.
The thing is, using these tools is just being lazy and being rewarded for it. That's not how games work.

The trick, and the danger, with trading, once the full game is in place, is that the more highly profitable routes, i.e. the illegal ones, the longer ones, carry with them flight danger, i.e. Personal Weapons into Surya, Gold from Freeport, or Palladium from Anderson's Escape, which are either illegal goods, or anarchic and, in the case of AE, far away by SC, and with a risk of multiple interceptions, so once interceptions get scaled up based on rank, for example, it won't be the cakewalk it is today.

Plus, longer routes, once the system opens up to hundreds and thousands of stations, will still need to be manually computed/analysed.

Anybody that's actually used the tools will also be able to tell you that the big question, the Any to Any system query, is extremely slow and takes upwards of 15-20 minutes to complete as-is, with just 55 star systems, so the tools aren't the destroyer of worlds they're made out to be, some sense is required to make optimal use of them without also waiting 10-15 minutes each time. The tools also can't distinguish between illegal and legal, and because of the dynamic economy, the answer isn't always the same either, so the question must be asked regularly.


So, just because the tools can calculate an A to B answer, doesn't mean that they're the be all end all of the problem, it's just the beginning. Sure, the specific A to B route is profitable, but they're not going to be effective in computing A to B to C to D going forward as the full galaxy is too big to do that in a reasonable timeframe, and also the other question of "can you survive it". Even with the tools, i still need to check the galaxy map to ensure I'm not off on a wild goose chase.

Additionally, trading Fish, for example, no matter how profitable, is supposed to be the slow way to earn anyway, there's next to no flight risk (aside from docking comedy, that is ;)).
 
Last edited:
I made a screenshot of the tradewindow and used vb.net to filter out the noise.. with the fixed font it shouldn't be long before i can scrape the data with resonable accuracy.

Yes i would have to take 3 screenshot to capture all data, but it beats
a manual spreadsheet input.

2el5u90.jpg

http://nl.tinypic.com/r/2el5u90/8
 
Last edited:
  • Like (+1)
Reactions: Ian
I made a screenshot of the tradewindow and used vb.net to filter out the noise.. with the fixed font it shouldn't be long before i can scrape the data with resonable accuracy.

Yes i would have to take 3 screenshot to capture all data, but it beats
a manual spreadsheet input.

Excellent - check further up for IxForres's post some tools to change the perspective format back to normal.
I look at the freely available tesseract OCR program, which you can train to recognise new fonts. Running in the default mode it made a terrible job, but looking at the training data it had actually recognised most of the characters on the screen - so not sure whats going on there.
 
Last edited:
Been searching with google and the like, cant find this announcement anywhere, does someone have the link to he original announcement?
 
FD: "These tools messes up our market algorithms so the game is not enjoyable. Thus, we are forbidding them, but may think of an API that can be used to produce output that doesn't completely mess up the algorithms."

Modders: "Screw you, I do what I want, and I don't care if it destroys the game for lots of people".


... seriously? Stop arguing, because the argument you present is a stupid argument. The game systems are built to emulate a real world, and you are taking measures to make it unreal. For what? Because you are too lazy to play the game the way it is meant to be played? Because you enjoy destroying the work of others? Because you like to detract from the joy others get who want to get "the real" experience? How can you possibly justify continuing, when you know FD are against it, and you know the reasons why?

Well, thats not the argument made by FD. Their argument is that it somehow affect client stability of those using the mods and they spend time debugging that information.
 
Please don't - duplication is very irritating.



Most likely no, but I've not decided yet. It'll depend on if anyone can actually be bothered typing numbers into the BPC - which is a high barrier to cross to start with. It won't happen multiple times per station visit (as you'll likely do it only once) and as it's not automatic I'd expect most people not to bother, so the firehose becomes less attractive compared to when it was used with the marketdump which was repeatedly squriting high volumes of data around. However, that remains to be seen.

It can be done sans firehose with a back end database - the performance of the amazon database I've been using has been negligible, even at max load. Having the firehose just adds another layer to sending the info to the DB directly - it's duplication on the firehose and listener side. If people want to query the amazon database, I can allow that. Should the traffic get too much then a firehose might be worth it - but with manual entry, most users won't bother and I'd expect the BPC to turn niche instantly.

Well if getting people to type in data, how about adding a little incentive for manually entering data. Like not beeing able to sync market data until they have submitted updated numbers for x systems. That would pretty much force anyone using your tool also contribute to the data it uses.
 
Well if getting people to type in data, how about adding a little incentive for manually entering data. Like not beeing able to sync market data until they have submitted updated numbers for x systems. That would pretty much force anyone using your tool also contribute to the data it uses.


id type in numbers as it gives the the max known profit...which even with trial and error manually will be impossible given the number of systems to come
 
Well, IMO there is no point in using existing OCR, since it's a known consistent font and thus can very quickly be done (just by using shape matching) based on said font (yes I'm simplifying the algorithm vastly)

I may if I get chance over the weekend have a quick play to see what can be thrown together, but I expect it will be quite simple to interpret the data.

That was my idea too. Maybe even use a phone app for taking the picture and feeding it to google doc directly with a "ok" button and edit for any wrong readings.

Its pen and paper for 2014, a good tool for manual updating data like Theo had in first beta is good enough to get a feeling for the market.
 
Deskewed the image, started work on number recognition.

All the different image resolutions will be a problem i have to look at later,
just working on a proof of concept atm.

2087fva.jpg
 

Slopey

Volunteer Moderator
I've got it deskewed, and about 70% accurate for now, but I don't think it's going to be perfect, and it's going to be difficult for looking for invalid but reasonable data.

As I'm trying to get a solution out the door sooner rather than later (to coincide with Beta 2) I'll likely add a manual input function into the BPC for now, but I have a few other ideas.
 
I've got it deskewed, and about 70% accurate for now, but I don't think it's going to be perfect, and it's going to be difficult for looking for invalid but reasonable data.

As I'm trying to get a solution out the door sooner rather than later (to coincide with Beta 2) I'll likely add a manual input function into the BPC for now, but I have a few other ideas.

All Hail Slopey :) looking forwards to your beta 2 stuff :)
 
Top Bottom