How to scrape LinkedIn, Reed and Totaljobs jobs in one run with Python
If you look for data analyst jobs in Manchester, you end up with four tabs open: LinkedIn, Reed, Totaljobs and maybe Welcome to the Jungle. Each has its own filters and salary format, and th...
Résumé
If you look for data analyst jobs in Manchester, you end up with four tabs open: LinkedIn, Reed, Totaljobs and maybe Welcome to the Jungle. Each has its own filters and salary format, and the same ad often shows up on two of them. I wanted one table instead, so I used the Job Scraper on Apify and ran a small search to see what comes back. This post walks through that run: the input, the fields you get, a no-code route and a Python route with pandas. The test run Input: country gb, query data analyst, location Manchester, jobs posted within the last week, max 12 jobs per search, company hiring signals on. What came back after about 20 seconds: 35 job records plus 1 company hiring signal record 12 from LinkedIn, 12 from Reed, 8 from Totaljobs, 3 from Welcome to the Jungle 16 of the 35 jobs had a published salary, all converted to yearly EUR 1 duplicate merged: a Jacobs senior data analyst role found on LinkedIn and Totaljobs became one record with an alsoFoundOn link For the UK the actor picks LinkedIn, Reed, Totaljobs and Welcome to the Jungle by itself. You only choose the country. What data you get Every board returns the same schema. These are the fields I used most: Field Example from the run Notes source reed Board the job came from title Revenue Operations Analyst As posted company BBC Employer or agency name city Salford Parsed from the location text remote hybrid remote, hybrid, onsite or empty if unknown salaryMin / salaryMax 40000 / 45000 Original numbers salaryCurrency / salaryPeriod GBP / year Also day or hour salaryYearlyMinEur 47858 Converted to yearly EUR skills ["SQL", "Power BI", "Tableau", "Excel"] Detected from the text daysSincePosted 1 Freshness alsoFoundOn list of other boards Filled when duplicates are merged url link to the ad Always present One thing I liked: a Totaljobs contract listed at 486 GBP per day came out as salaryYearlyMinEur: 127924. Day rates and annual salaries end up comparable in one column, which saves a lot of regex. Skills and remote status are thinner without full details. In my run, remote was set for 14 of 35 jobs. If you need full descriptions and better skill detection, turn on includeDetails (it costs extra, see pricing below). Main input options Country picks the boards. Poland gets Pracuj.pl, JustJoin.it, No Fluff Jobs and LinkedIn; Germany gets LinkedIn, StepStone and XING. Job titles or keywords and Locations are lists. Every query runs against every location on every board. Leave locations empty to search the whole country. Job boards lets you override the country preset, for example only Reed and Totaljobs. Posted within, Workplace, Employment type and Experience level are the usual filters. Some boards support them natively, others get filtered after scraping. Min. yearly salary (EUR) and Only jobs with salary filter on the converted value. Title must include and Exclude companies clean up noise. Max. jobs per search caps volume and cost. Company hiring signals adds one summary record per company that has at least minJobsPerCompany jobs. Only new jobs with a Monitor name returns only jobs not seen in earlier runs. Two lessons from my run. First, keyword search pulls in neighbours: Welcome to the Jungle returned a product manager and a cyber security role for "data analyst". titleMustInclude: ["analyst"] fixes that. Second, one training provider posted 5 near-identical trainee ads for Bolton, Stockport and UK-wide locations. The hiring signal flagged it as "high" velocity, which is technically true and useless for my purpose. excludeCompanies handles it. Option 1: no code Open the actor page and click Try for free. Set Country to United Kingdom. Type data analyst in Job titles or keywords and Manchester in Locations. Set Posted within to Last week and Max. jobs per search to something small like 12 for a first test. Click Start. A run like mine takes well under a minute. Open the Jobs view in the dataset and export to CSV, Excel or JSON. The dataset also has ready views for job details, company hiring signals and monitoring results. Option 2: Python with apify-client and pandas Install the two packages: pip install apify-client pandas Then run the actor and load the results: from apify_client import ApifyClient import pandas as pd client = ApifyClient("YOUR_APIFY_TOKEN") run_input = { "country": "gb", "searchQueries": ["data analyst"], "locations": ["Manchester"], "postedWithin": "week", "maxJobsPerSearch": 12, "titleMustInclude": ["analyst"], "dedupe": True, } run = client.actor("flarewow/job-scraper").call(run_input=run_input) items = client.dataset(run["defaultDatasetId"]).list_items().items df = pd.DataFrame(items) jobs = df[df["recordType"] == "job"].copy() print(jobs["source"].value_counts()) paid = jobs.dropna(subset=["salaryYearlyMinEur"]) print(f"{len(paid)} of {len(jobs)} jobs show a salary") cols = ["source", "title", "company", "city", "remote", "salaryYearlyMinEur", "salaryYearlyMaxEur", "url"] paid.sort_values("salaryYearlyMinEur", ascending=False)[cols].head(10) jobs[cols].to_csv("manchester_data_analyst_jobs.csv", index=False) call() waits for the run to finish, so the dataset is complete when you read it. If you add companySignals: True, filter on recordType == "company-signal" to get the company summaries in a separate frame. A quick look at salaries from my run: the lowest published minimum was 29,911 EUR (two trainee and junior roles), the highest annual one 71,787 EUR, and the day rate contract sat far above both. Small sample, but enough to see that "data analyst" in Manchester spans junior trainee schemes to senior contract work. Scheduling and integrations For a daily job alert, save the input as a task, set onlyNewJobs: true and a monitorId like manchester-data-analyst, then add a schedule in Apify Console. Each run returns only ads it has not seen before, and those older ads are not charged again. From there you can push results to Google Sheets, Slack, Make, Zapier, n8n or a webhook through Apify integrations. The actor also works as a tool through the Apify MCP server if you want an AI agent to run searches in plain language. Pricing The actor uses pay-per-event pricing. Per 1,000 results, from the current price list: Event Free plan Business plan Job from a job board $2.50 $1.80 Job from a company career site $6.00 $4.50 Full job details add-on $1.00 $0.70 Indeed / Glassdoor premium job $4.50 $3.50 Company hiring signal $0.01 each $0.007 each My run charged 35 job events and 1 company signal, which is roughly 10 cents at Free plan prices. You can set a maximum cost per run in the run options and the actor stops when it hits the limit. Responsible use The actor reads public job postings without logging in. Job ads name companies, which is fine for analysis, but some boards include recruiter names inside descriptions. If you store full descriptions, check whether you need them, and keep GDPR in mind if you contact anyone. Respect each site's terms, keep volumes reasonable and do not republish ads as your own job board without permission. This tutorial was written with AI assistance and checked by the author. All numbers come from a real run of the scraper.