Senior Software Engineer, Web Crawling Platform at HubSpot


Title: Senior Software Engineer, Web Crawling Platform (Remote US)

Location: Remote – USA

We are growing the Web Crawling Platform team and are looking for a Senior Software Engineer that can help us build a world-class web crawling infrastructure. The goal of the team is to build a web data platform that crawls and stores a large portion of the internet and extracts data and intelligent insights from discovered web pages. The challenges you’ll face on the web crawling platform team include:

  • Scaling the crawling platform to support to millions of daily domain and single page crawls
  • Designing elegant yet practical APIs that can support dozens of use cases that need data from web crawls
  • Building and expanding on the distributed storage systems to efficiently persist and retrieve terabytes of information extracted from crawled web data
  • Further expand our data parsing capabilities to extract more useful information and insights from web data in collaboration with our Machine Learning team

We are looking for people who:

  • Can help us realize the vision of making our web crawling more powerful, reliable and highly performant for HubSpot application developers that use it to power dozens of use cases and counting.
  • Are willing and able to work with the user base, in this case, other developers, to define and build tools that make the lives of application developers easier and their tools more reliable.
  • Are interested in working with and supporting infrastructure at scale.
  • Able to approach complicated technical problems and analyze tradeoffs with empathy for the developers, yet building creative solutions.
  • Able to demonstrate pragmatic decision making and problem-solving abilities.

Bonus points for:

  • Being someone with experience on web crawling at scale.

Confidence can sometimes hold us back from applying for a job. But we’ll let you in on a secret: there’s no such thing as the ‘perfect’ candidate. HubSpot is a place where everyone can grow. So however you identify and whatever background you bring with you, please apply if this is a role that would make you excited to come into work every day.

See all Developer Jobs >

Sign up for Daily Remote Job Alerts!

Want Access to 25,000+ More Remote and Flexible Jobs?

More Jobs

More Jobs

Part-time to full-time,
freelance to employee

More Career Fields

More Career Fields

50+ flexible
job categories

More Resources

More Resources

Q&A's, webinars,
career coaching & more

Learn More About Our Premium Service