Axis Information
Here we can set each of the three axes to explore the relevant job listings. The job listings are evaluated against the cosine distance to these 3 axes.
X Y Z Each of the axis to search along
N The number of interval to search along the axis
K The number of points to return at each interval
You can set the axes to whatever you would like, but keep in mind that this dataset is based around job listings in Machine Learning and AI.
After clicking the "Update" button, each of the three axes will be embedded using an LLM to generate a vector. This vector is then used to search the vector database to find relevant job listings.
For more information, please see the info tab at the top of the app.
Axis Information
This section provides more information about a specific job listing selected in the plot above. In the 3 progress bars, the cosine similarity is shown to represent how closely this specific job listing aligns with the given axis.
Using the cosine similarity here is different from the plot above, which is the cosine distance. This provides a more intuitive representation at a glance, as higher means more aligned. These cosine similarities have also been normalized to the ranges of distances on the chart.
The axis summary below, takes the RAG queries defined above for the axis, and then compares the job on each of these axis to generate this alignment summary
Correlations
The axis correlations represent the degree of linear relationship between the different axes. A correlation value of 1 indicates a perfect positive correlation, -1 indicates a perfect negative correlation, and 0 indicates no correlation.
Let's have a look at two correlations from the initial plot, "Machine Learning Engineer vs Data Science" and "Machine Learning Engineer vs Accountant"
Correlation
0.81
This 2-axis plot of the cosine distance of the job listing against "Machine Learning Engineer" and "Data Scientist" clearly shows there is a positive correlation between the two terms. In relevance to what we are measuring here, that means that as a Job Listing becomes closer to a "Machine Learning Engineer" it also becomes closer to a "Data Scientist". This makes intuitive sense as they are very closely related roles, with some minor differences.
Correlation
-0.21
In contrast to the plot above, we can see that there is no real significant correlation between the roles of a "ML Engineer" and an "Accountant". This again makes intuitive sense, as these roles are a lot less related than those above.
This information is represented easily in the correlation bars on the main page
Plot Summary
Here is where we can generate a meta-analysis of the three separate axes using Weaviate's built-in generative search.
This search will happen in two stages. In the first stage, we select the job that is the closest on each axis, and we use that vector to perform a "near vector generative search" using a prompt that we have given it to summarize the information.
Here is an excerpt of the prompt and the call to Weaviate:
prompt = """... Please provide a concise summary that highlights the key aspects of these job listings
that contribute to their alignment with the "{text}" axis. Consider the following points in your summary:
1. What are the common themes, skills, or requirements mentioned across these job listings?
2. How do these job listings relate to the concept of "{text}"?..."""
single_query = listings.generate.near_vector(near_vector=job_weaviate.vector['default'], limit=3, grouped_task=prompt)What we receive out of Weaviate will be 3 summaries on job listings that closely relate to the specified axis.
Machine Learning Engineer
Summary: The key aspects that make these job listings aligned with the "Machine Learning Engineer" axis include a focus on developing AI models, natural language understanding (NLU), natural language processing (NLP), and machine learning (ML) algorithms. The job listings require experience in Python, cloud platforms ...
Data Scientist
Accountant
These three summaries are then put into a meta-analysis prompt that is formatted like this:
Based on the summaries of the top 3 job listings for each axis, provide an overview of the similarities and differences between the axis:
Provide you answer in this specific format: ...
This output of this last prompt is displayed below and should give an insight into the similarities and differences between the axis