To add: I can provide information about the data security practices of the tools, including details on data sharing, any identified security vulnerabilities, and their access to sensitive data.

Yes. The outcome can be used as a ground truth for Ragas, Langchain/Llamaindex evals.

Hi,

While we call LLMs(internal and external, based on instruction type), the output generated by LLMs can't be taken as ground truths unless we do rigorous evaluations. We have our own metrics when it comes to what could be called a ground truth, based on the user's seed information and business logic. Accuracy & preciseness needs also differ from use-case to use case. Function calling adds in another layer.

Another value add is type of instructions that we can generate. We expose 7 currently, and are working on exposing more instruction types. The challenge is to create ground truth of wide variety of cases that a given user can ask for a business including guardrailing.

We have built internal tools and agents to solve for those, and are internally discussing the ideal way to expose it to users, and whether it would be beneficial for the community. Any thoughts on that would be appreciated.

Automation took a significant amount of time for us as well, so at scale, even a reliable automated CI/CD pipeline is indeed a value add in itself.

Lmk if I can add more details to answer the question.

Can you elaborate on the use case a bit? HTML as a part of the prompt for what kind of use case.

I am scraping some information from a list of company's website. I would like to create a evaluation set for my agent.

How would this cover for security vulnerabilities and other invisible tech debt that was incurred in the previous iterations? Generally, in the migrations, humans discard a lot of old code for new modules and capabilities which have become available. Does the AI take care of that?

This is much needed. For someone looking to quickly implement, having a simple interface goes a long way.

Thanks