HTML Data Storage for Enrichment

Claygent cannot efficiently scrape HTML. Instead, use an HTTP API call.

Setup Requirements

• Configure the API call as GET
• Normalize the website column to use HTTPS domain URLs (required for configuration)
• Add the API response as a new column by clicking into the cell value

Limitations to Note

• Some websites don't support HTTPS
• Some websites may be unavailable
• Cell data may exceed size limits

Storing HTML Data for Token Efficiency

Your approach makes sense. Here's what you're proposing:

1. Scrape website pages
2. Store extracted information in a JSON column
3. Use that column as input instead of {{ website }}

Benefits

• Avoids repeated scraping of the same company pages during testing or re-enrichment
• Reduces token consumption by referencing stored data instead of re-processing raw HTML
• Improves performance on subsequent enrichment operations

This strategy is particularly useful when you need to test or iterate on your enrichment logic multiple times without incurring the costs of re-scraping and re-tokenizing the same source data.

NULL

The original answer provided is actually a question rather than an answer. It poses a query to the community about whether storing HTML in a column to save tokens during enrichment is a best practice, but it doesn't provide any actual guidance, recommendations, or actionable information to reformat.

To store HTML data in a column during enrichment, consider your intended use case for the data.

Recommended approach:

• Use JSON.stringify() in a formula column to convert the HTML data into JSON format
• This allows you to structure and store the output efficiently

Before implementing, clarify what you plan to do with the stored data, as this will determine the most appropriate storage and formatting method.

Storing HTML Data in a Column

Storage Limitations

• HTML data stored in a new column is limited to 8KB, which is typically sufficient
• This limitation makes it reasonable to let Claygent handle scraping directly rather than pre-storing the data

Benefits of Direct Scraping

• Claygent can automatically access related websites if required information isn't found on the original source
• Avoids the need to manage and update stored HTML data manually