Discussion Has any one experimented with OCR tools for scraping the commodities market prices?

Any commentary to go with this. What software, etc?

I'm using gocr as I said. If any one wants to collaborate we could put it on one of the public repositories to work on and that would be great. I'm trying out on linux first, but I'm sure it could all be packaged up for windows pretty easily.
 
I'm using gocr as I said. If any one wants to collaborate we could put it on one of the public repositories to work on and that would be great. I'm trying out on linux first, but I'm sure it could all be packaged up for windows pretty easily.

I was replying to Nog but you got in there first.
 

wolverine2710

Tutorial & Guide Writer
Wgt2fwu.png

OoxCEfW.png

24N2NYD.png

Nog. That looks like a pretty good Proof Of Concept. Can you share how you did this - I know someone else asked it also. I´ve noticed you OCR only the first colums. Is there a reason for that?
 
After a bit of clean up and some simple corrective substitutions (CON SUMER -> CONSUMER sort of thing) I can now get this.
So from this
rYliIJP.jpg
Code:
DIESEL SURVEY                                      
LARGE POPLLATION TOURISM ECONOMY (FEDERATION                           
HYDROGEN FUEL        182       -  136179 MED       ,,   - 189 C
CLOTHING             393       -      84   -        -   - 303 CR
CONSUMER TECHNOLOGY  7297      -       7   -        -   - 6757 CR
DOM. APPLIANCES      632       -    1424 MED        -   - 552 CR
BEER                 236       -    3856 MED        -   - 1
ANIMAL MEAT          1471              - MED        -   - 1332 R 
COFFEE               1471      -       8 MED        -   - 1367 CR
FISH                 792       -    1137 MED        -   - 642 CR
FOOD CARTRIDGES      202       -     206 MED        -   - 127 C
FRUIT AN_ VEGETABLES 393       -    2490 MED        -   - 328 CR
GRAIN                271       -    1989 MED        -   - 213 CR
SYNTHETIC MEAT       322       -     215 MED        -   - 254 CR
TEA                  1659      -     357 MED        -   - 1470 CR
BASIC MEDICINES      473       -    2490 HIGH        -   - 317 C
You can see it struggles in a couple of cases, Animal Meat for e.g.

It depends on the background image somewhat, as I can also get misaligned columns, but I think by also scanning across the image and correlating with the down direction I may be able to double check results and at least discard bad data.
 

wolverine2710

Tutorial & Guide Writer
That ALSO looks very promising. Perhaps not directly usable but Slopey in the BPC backend has/had quite a lot of checks to make sure false/garbled data could be detected. Perhaps he can share those. Also there a approximate string routinesout there which could guess what a string is ment to be.

Lets just assume everything works 100% - knocking wood. What must a user install on his/her PC to get the market info.and what actions? Info which could be used for eirther local mode or uploading to some kind of phoenix EMDN?

Note: Did the new UI of SB2 make things easier?
 
They'll probably never be able to stop this kind of thing, and that's unfortunate, but it's more unfortunate that people do it in the first place. The reality is that it leads to a diminishment of the actual gameplay dimension of trading simply in order to provide a shortcut.

The trade system is obviously not inherently where it needs to be right now, though I think it's actually much better than beta 1 in terms of direction, but they've chosen to emphasise and de-emphasise specific things in each release, and rebalance the game progression as they go. Presumably we aren't looking at the final model.

When we get there, the ideal is that if you opt to be a trader, you actually try to use the system as it is to generate good trades for yourself, rather than relying on a website to tell you where to go.
 

wolverine2710

Tutorial & Guide Writer
It seems I've made a mistake when I created this thread. This time I forgot something essential.
I've added it to the OP and I'm posting it beneath.


Please do NOT turn this into another pro vs contra third party (trading) tools thread. Those discussions for me are the equivalent of the trenches war of WWI. Nobody makes progress. If you want to discuss it, the perfect place is "Direct question for Michael Brookes". . I hope and trust you respect my wishes.
 
Last edited:
As another commander said, it is a proof of concept, so it only showed the first two columns, once you have success with them, add the rest is easier.
I used tesseract trained (train tesseract is a bit complicated, but can be done, and it is very important for success) with pre-segmented images. Whole process is done in a single step and result is (will be) a xls file with the data.
When I have something worth sharing, if no objection from frontier, of course I will.

With extracted data, next step is calculate best routes, etc

 
After a bit of clean up and some simple corrective substitutions (CON SUMER -> CONSUMER sort of thing) I can now get this.
So from this
rYliIJP.jpg
Code:
DIESEL SURVEY                                      
LARGE POPLLATION TOURISM ECONOMY (FEDERATION                           
HYDROGEN FUEL        182       -  136179 MED       ,,   - 189 C
CLOTHING             393       -      84   -        -   - 303 CR
CONSUMER TECHNOLOGY  7297      -       7   -        -   - 6757 CR
DOM. APPLIANCES      632       -    1424 MED        -   - 552 CR
BEER                 236       -    3856 MED        -   - 1
ANIMAL MEAT          1471              - MED        -   - 1332 R 
COFFEE               1471      -       8 MED        -   - 1367 CR
FISH                 792       -    1137 MED        -   - 642 CR
FOOD CARTRIDGES      202       -     206 MED        -   - 127 C
FRUIT AN_ VEGETABLES 393       -    2490 MED        -   - 328 CR
GRAIN                271       -    1989 MED        -   - 213 CR
SYNTHETIC MEAT       322       -     215 MED        -   - 254 CR
TEA                  1659      -     357 MED        -   - 1470 CR
BASIC MEDICINES      473       -    2490 HIGH        -   - 317 C
You can see it struggles in a couple of cases, Animal Meat for e.g.

It depends on the background image somewhat, as I can also get misaligned columns, but I think by also scanning across the image and correlating with the down direction I may be able to double check results and at least discard bad data.

It's gotten easier with Beta 2 since the market window is already in a correct perspective, but the problem is actually the buy column as it uses a gray that is difficult to distinguish from the background.
 
However, they can change the EULA and make you agree to it upon installation, like every other piece of software does today, surely?

In most countries there are restrictions to what is allowed in an EULA or not (you can always write it down and the customer may accept it but a court will say that passage XY is not valid or even illegal and therefore does not have to been followed)

And takign Screenshots and extracting the text from them is certainly one of the things that can not been forbidden in ost countries.
 

wolverine2710

Tutorial & Guide Writer
It's gotten easier with Beta 2 since the market window is already in a correct perspective, but the problem is actually the buy column as it uses a gray that is difficult to distinguish from the background.

A viable route seems to be doing preprocessing with imagemagick or graphicsmagick (fork of imagemagick). See also my post number 43. Some preprocessing info can be found here. Googling for "imagemagic ocr precprocessing gives lots of results like: ImageMagick to preprocess image for tesseract-ocr
or OCR using Tesseract and ImageMagick as pre-processing task. Both imagemagick and graphics have LOTS of commandline options and can controlled using multiple programming languages. Unfortunately still on holdiday so can´t try things out.

NOTE: It seems at least two commanders are getting pretty good results with OCR. Thanks for the hard work and ofc everyone who has contributed to this thread. Really appreciated!! The OCR route seems quite viable and do able.
 
I suppose it's the difference between those who trade as an end in itself, and those who trade to earn credits to buy more ships/equipment, a different sort of end. I'm in the former camp, and enjoy the slow plotting and exploring of routes, keeping notes, finding bargains, etc. I imagine that the time spent on data scrapping could be used just as productively in-game trading.

I do recall in the original 1984 game you just went by broad system economy types - and I'd say anyone using that approach here across the five main economy types won't go far wrong. I keep reading on the boards that the current limited goods to trade is a cunning FD plan to get us running more missions. So just make sure you're trying to solve the right problem. i.e. current trading conditions might only last until Beta 3 at the end of the month. By the time it's 'solved' with this OCR route, the issue may be gone. Remember, they still need to test the demand side of trading, so I think trading will be back in play soon enough.

If people still struggle with trading once it goes to a normal level, so be it. I struggle with fighting. Both are skills. Both need practice. Can I have an OCR screen-tool to help me fight better please? Maybe an automated programme on my laptop that I can toggle on to do most of my targeting/firing for me, while I try to straighten up and fly right.
 
FD: "These tools messes up our market algorithms so the game is not enjoyable. Thus, we are forbidding them, but may think of an API that can be used to produce output that doesn't completely mess up the algorithms."

...

How can you possibly justify continuing, when you know FD are against it, and you know the reasons why?

Don't be naive.
The first thing which happens when it get's released is people building such systems if it doesnt already exists.
It won't help when FD says:
" ...but ..it's forbidden because our algos are not working!"

It's certainly not liked but they have to forseen that such a system will be build the one way or the other.

What I wonder WHY is the algo not working?

My experience from Beta 1 is that it is done wrong. If you work with demand and supply the price movement can't stop on the galatic average.

I don't know if it has changed in Beta 2 ..hardly have time to play at the moment. Last I read was that the market is dead in Beta 2 anyway.

I truely thing you can come up with an algo that will work with or without such a tool if you think enough.
 
The balance (apologies to the OP for going with the theme the thread has run down, rather than their intended one) is between allowing players who just check the in-game system getting a good enough experience and those who want to put a bit more effort compiling a set of useful data to reference getting enough out of it.

It's ridiculous to ask that someone just dipping into trading now and then and not keeping any kind of record should be able to make the same sort of money that someone who's actually dedicated to trading, takes note of patterns, fluctuations and underused routes could make. If you put the time in, you should be rewarded.

As to a tool which allows multiple people to benefit from a particularly lucrative trade route? If the market is ever going to be robust enough to actually make consistent sense, then it will never matter how many people run one given trade route, it will just devalue that trade route and people will have to start shipping other commodities. That's not a bad thing, it's a good thing. We want as diverse a portfolio of commodities being shipped as possible.
The main problem with the trading system right now is that there's very little reason to ever trade low-value commodities. I see some effort to counter that with the missions added in Beta 2, but it's not sufficient (as the value of the mission is relative to the value of the commodity requested).
 

wolverine2710

Tutorial & Guide Writer
I suppose it's the difference between those who trade as an end in itself, and those who trade to earn credits to buy more ships/equipment, a different sort of end. I'm in the former camp, and enjoy the slow plotting and exploring of routes, keeping notes, finding bargains, etc. I imagine that the time spent on data scrapping could be used just as productively in-game trading.

I do recall in the original 1984 game you just went by broad system economy types - and I'd say anyone using that approach here across the five main economy types won't go far wrong. I keep reading on the boards that the current limited goods to trade is a cunning FD plan to get us running more missions. So just make sure you're trying to solve the right problem. i.e. current trading conditions might only last until Beta 3 at the end of the month. By the time it's 'solved' with this OCR route, the issue may be gone. Remember, they still need to test the demand side of trading, so I think trading will be back in play soon enough.

If people still struggle with trading once it goes to a normal level, so be it. I struggle with fighting. Both are skills. Both need practice. Can I have an OCR screen-tool to help me fight better please? Maybe an automated programme on my laptop that I can toggle on to do most of my targeting/firing for me, while I try to straighten up and fly right.

I would like to respond but can´t. You seem to have missed my OP request. I repost it here before we get into another discussion - which has just started again.

Please do NOT turn this into another pro vs contra third party (trading) tools thread. Those discussions for me are the equivalent of the trenches war of WWI. Nobody makes progress. If you want to discuss it, the perfect place is "Direct question for Michael Brookes". . I hope and trust you respect my wishes.
 
Last edited:
I would like to respond but can´t. You seem to have missed my OP request. I repost it here before we get into another discussion - which has just started again.

Please do NOT turn this into another pro vs contra third party (trading) tools thread. Those discussions for me are the equivalent of the trenches war of WWI. Nobody makes progress. If you want to discuss it, the perfect place is "Direct question for Michael Brookes". . I hope and trust you respect my wishes.

I can see why you wish for that, and don't want to derail your OP entirely, but trying to keep on-topic I think my middle paragraph is pertinent, asking the question "what problem is being solved?". If it's the scant trading in Beta 2, that situation may be resolved in three weeks' time with Beta 3, or later once the game properly gets going. I'd say first define the problem, then find a solution. Are people that sure that trading will be so challenging in the live game that going to the effort of OCR screenscraping will really be worth it?

Looking back to your OP subject line, I presume this was answered the first time another poster replied "yes". :)

Good luck with which ever way you go. I merely observe that a problem in B2 may no longer be an issue in B3, B4, Bn, or the live game.
 
Good luck with which ever way you go. I merely observe that a problem in B2 may no longer be an issue in B3, B4, Bn, or the live game.

And if it isn't, great.

As a bonus we have learnt something. I didn't know much about OCR. I didn't know much about how P2P in games operated, but I found there is a large body if academic work on it.

Just because there is nothing to show for it, the time hasn't been wasted if you learnt something from it.
 

wolverine2710

Tutorial & Guide Writer
Was on a holiday, back since a few days and checked my threads. Some very promising work has been done so far and some proof of concepts were shown. To those commanders:
Have you been able to progress further and get all data from the commodities market?
Could you share your POC with us?
For those who are new to this thread, perhaps you see a challenge and want to try it as well?

The way I see it. We first need the OCR part done first. After that we need ways to automize it and a way to send it to a new phoenix EMDN or a way to use that data in local mode for for example the BPC or Trade Dangerous by kfsone. This does NOT have to be done by the same commander!!! The load can be spread. We at this point ONLY need a good and reliable OCR solution.I know it has been done for other websites/project. I'm really hoping we get a solution for ED as well.
 
I was working on a tray app that hooks F11 to take a screen capture and passes that on to tesseract. That would then upload that to a website in json.

This site will generate stations based on names from the capture and ask the players to occasionally ask for distances to stars to crowdsource system coordinates.

So far I have a tray app that takes captures the screen but if I get time this evening I will look at putting in the tesseract.
 
Top Bottom