Skip to content

Commit bd29240

Browse files
nelly-hatevanelly-hateva
committed
Merge remote-tracking branch 'origin' into Statnett-255
2 parents 227509f + 4b97d30 commit bd29240

File tree

12 files changed

+533
-559
lines changed

12 files changed

+533
-559
lines changed

‎CHANGELOG.md‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,17 @@
1-
Next release
1+
1.4.0-rc1
22
============
33

44
* [#255](https://github.com/statnett/Talk2PowerSystem_PM/issues/255): OBO auth flow for Cognite
5-
* [#255](https://github.com/statnett/Talk2PowerSystem_PM/issues/255): Update the version of `cognite-sdk` from `7.86.0` to `7.89.0`
5+
* [#255](https://github.com/statnett/Talk2PowerSystem_PM/issues/255): Update the version of `cognite-sdk` from `7.88.0` to `7.89.0`
6+
7+
1.3.0-rc1
8+
============
9+
610
* [#256](https://github.com/statnett/Talk2PowerSystem_PM/issues/256): Update the queries for the ontologies and datasets information served from the `__about` endpoint
711
* [#251](https://github.com/statnett/Talk2PowerSystem_PM/issues/251): Change N-Shot tool configuration to default to the base GraphDB
8-
* [#251](https://github.com/statnett/Talk2PowerSystem_PM/issues/251): Change N-Shot tool configuration SPARQL query template
12+
* [#276](https://github.com/statnett/Talk2PowerSystem_PM/issues/276): Change N-Shot tool configuration SPARQL query template, so that it outputs unique SPARQL queries
913
* [#254](https://github.com/statnett/Talk2PowerSystem_PM/issues/254): Update the version of `ttyg` from `1.9.3` to `1.10.0`, so that the chat bot can run without admin access to GraphDB
14+
* [#254](https://github.com/statnett/Talk2PowerSystem_PM/issues/254): Update the version of `cognite-sdk` from `7.86.0` to `7.88.0`
1015

1116
1.2.0-rc4
1217
============

‎README.adoc‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -98,7 +98,8 @@ Steps to create an official release (from the main branch):
9898

9999
. `conda activate Talk2PowerSystemLLM`
100100
. `poetry version <major|minor|patch>`.
101-
. `git add pyproject.toml`
101+
. Update `CHANGELOG.md`
102+
. `git add pyproject.toml CHANGELOG.md`
102103
. `git commit -m "Bumping version from <previous-version> to <current-version>"`
103104
. `git push -u origin main`
104105
. Create a release from https://github.com/statnett/Talk2PowerSystem_LLM/releases[the GitHub interface]. The tag and the release title must match the version from poetry!
@@ -115,7 +116,8 @@ For example, if the current version is `1.2.0-rc1`, the next version must be `1.
115116
If the current version is a development version, then the next pre-release version must follow the semantic versioning convention on how to increment the major,
116117
minor and patch parts of the version and add `-rc1` at the end.
117118
For example, if the current version is `1.1.0-dev0` and the next release will be a major one, the next version must be `2.0.0-rc1`.
118-
. `git add pyproject.toml`
119+
. Update `CHANGELOG.md`
120+
. `git add pyproject.toml CHANGELOG.md`
119121
. `git commit -m "Bumping version from <previous-version> to <current-version>"`
120122
. `git push -u origin main`
121123
. Create a release from https://github.com/statnett/Talk2PowerSystem_LLM/releases[the GitHub interface]. The tag and the release title must match the version from poetry!

‎config/dev+retrieval.yaml‎

Lines changed: 15 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -32,18 +32,22 @@ tools:
3232
PREFIX retr: <http://www.ontotext.com/connectors/retrieval#>
3333
PREFIX retr-index: <http://www.ontotext.com/connectors/retrieval/instance#>
3434
PREFIX qa: <https://www.statnett.no/Talk2PowerSystem/qa#>
35-
SELECT ?question ?query {{
36-
[] a retr-index:{connector_name} ;
37-
retr:query "{query}" ;
38-
retr:limit {limit} ;
39-
retr:entities ?entity .
40-
?entity retr:score ?score;
41-
qa:question ?question.
42-
?template qa:paraphrase ?entity;
43-
qa:querySparql ?query.
44-
FILTER (?score > {score})
35+
SELECT (REPLACE(GROUP_CONCAT(?q; separator="@"), "(.*?)@.*", "$1") AS ?question) ?query {{
36+
SELECT ?q ?query ?score {{
37+
[] a retr-index:{connector_name} ;
38+
retr:query "{query}" ;
39+
retr:limit 100;
40+
retr:entities ?entity .
41+
?entity retr:score ?score;
42+
qa:question ?q.
43+
?template qa:paraphrase ?entity;
44+
qa:querySparql ?query.
45+
FILTER (?score > {score})
46+
}}
47+
ORDER BY DESC(?score)
4548
}}
46-
ORDER BY DESC(?score)
49+
GROUP BY ?query
50+
LIMIT {limit}
4751
llm:
4852
azure_endpoint: "https://statnett.openai.azure.com/"
4953
model: "gpt-4.1"

‎config/ontology/cim-subset-pretty.ttl‎

Lines changed: 449 additions & 399 deletions
Large diffs are not rendered by default.

‎docs/AgentConfig.md‎

Lines changed: 18 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -38,18 +38,22 @@ tools:
3838
PREFIX retr: <http://www.ontotext.com/connectors/retrieval#>
3939
PREFIX retr-index: <http://www.ontotext.com/connectors/retrieval/instance#>
4040
PREFIX qa: <https://www.statnett.no/Talk2PowerSystem/qa#>
41-
SELECT ?question ?query {{
42-
[] a retr-index:{connector_name} ;
43-
retr:query "{query}" ;
44-
retr:limit {limit} ;
45-
retr:entities ?entity .
46-
?entity retr:score ?score;
47-
qa:question ?question.
48-
?template qa:paraphrase ?entity;
49-
qa:querySparql ?query.
50-
FILTER (?score > {score})
41+
SELECT (REPLACE(GROUP_CONCAT(?q; separator="@"), "(.*?)@.*", "$1") AS ?question) ?query {{
42+
SELECT ?q ?query ?score {{
43+
[] a retr-index:{connector_name} ;
44+
retr:query "{query}" ;
45+
retr:limit 100;
46+
retr:entities ?entity .
47+
?entity retr:score ?score;
48+
qa:question ?q.
49+
?template qa:paraphrase ?entity;
50+
qa:querySparql ?query.
51+
FILTER (?score > {score})
52+
}}
53+
ORDER BY DESC(?score)
5154
}}
52-
ORDER BY DESC(?score)
55+
GROUP BY ?query
56+
LIMIT {limit}
5357
cognite:
5458
base_url: https://statnett.cognitedata.com
5559
client_name: talk2powersystem
@@ -146,11 +150,11 @@ LIMIT {limit}
146150
- `tools.cognite.project` - OPTIONAL, DEFAULT=`prod` - Cognite Data Fusion project name.
147151
One of `dev1`, `dev2`, `dev3`, `test`, `prod` according to [CDF access from RNDP](https://github.com/statnett/Talk2PowerSystem_PM/wiki/CDF-access-from-RNDP).
148152
- `tools.cognite.client_name` - OPTIONAL, DEFAULT=`talk2powersystem` - Name of the client for logging purposes.
149-
- `tools.cognite.interactive_client_id` - OPTIONAL - If provided, interactive authentication is used (local run of Jupyter Notebook).
153+
- `tools.cognite.interactive_client_id` - OPTIONAL - If provided, interactive authentication is used (when you run on a dev machine the backend app with uvicorn or the Jupyter Notebook).
150154
Otherwise, `tools.cognite.token_file_path` or `tools.cognite.client_secret` must be provided.
151155
- `tools.cognite.tenant_id` - REQUIRED iff `tools.cognite.interactive_client_id` is present - Azure tenant ID. For example, `a8d61462-f252-44b2-bf6a-d7231960c041`.
152-
- `tools.cognite.token_file_path` - OPTIONAL - Full path on the disk to the cognite token file (run of Jupyter Notebook on RNDP). For example, `/var/run/secrets/microsoft.com/entra/cognite`.
153-
* `tools.cognite.client_secret` - OPTIONAL - Client secret for the Cognite confidential application (running from the backend app).
156+
- `tools.cognite.token_file_path` - OPTIONAL - Full path on the disk to the cognite token file (used when you run the Jupyter Notebook on RNDP). For example, `/var/run/secrets/microsoft.com/entra/cognite`.
157+
* `tools.cognite.client_secret` - OPTIONAL - Client secret for the Cognite confidential application (used for the backend app running on RNDP).
154158
155159
## `llm`
156160

‎pyproject.toml‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[project]
22
name = "Talk2PowerSystemLLM"
3-
version = "1.2.0-rc4"
3+
version = "1.4.0-rc1"
44
description = "Talk to Power System LLM"
55
authors = []
66
readme = "README.adoc"
@@ -51,5 +51,4 @@ build-backend = "poetry.core.masonry.api"
5151
[project.scripts]
5252
evaluation = 'talk2powersystemllm.scripts.run_evaluation:main'
5353
qa_dataset2rdf = 'talk2powersystemllm.scripts.qa_dataset2rdf:main'
54-
find_min_retrieval_limit_on_dev = 'talk2powersystemllm.scripts.find_min_retrieval_limit_on_dev:main'
5554
benchmark_graphdb_ttyg = 'talk2powersystemllm.scripts.benchmark_graphdb_ttyg:main'

‎src/jupyter_notebooks/Talk2PowerSystem.ipynb‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -111,7 +111,7 @@
111111
"\n",
112112
"\n",
113113
"conf = RunnableConfig(configurable={\"thread_id\": \"thread-123\"})\n",
114-
"messages = {\"messages\": [(\"user\", \"List timeseries\")]}\n",
114+
"messages = {\"messages\": [(\"user\", \"List all transformers within substation OSLO.\")]}\n",
115115
"last_message_id = run_agent(agent, messages, conf)"
116116
]
117117
},

‎src/talk2powersystemllm/app/server/main.py‎

Lines changed: 33 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,10 @@
7878
about_info: AboutInfo = None
7979
# noinspection PyTypeChecker
8080
agent_factory: Talk2PowerSystemAgentFactory = None
81+
# noinspection PyTypeChecker
82+
confidential_app: msal.ConfidentialClientApplication = None
83+
# noinspection PyTypeChecker
84+
COGNITE_SCOPES: list[str] = None
8185
ctx_request = ContextVar("request", default=None)
8286
LoggingConfig.config_logger(settings.logging_yaml_file)
8387
trouble_html = get_trouble_html()
@@ -87,6 +91,8 @@
8791
async def lifespan(_: FastAPI):
8892
logging.info("Starting the application")
8993
global agent_factory
94+
global confidential_app
95+
global COGNITE_SCOPES
9096

9197
redis_password = settings.redis.password
9298
redis_auth = ""
@@ -110,6 +116,18 @@ async def lifespan(_: FastAPI):
110116
settings.agent_config,
111117
checkpointer=memory_saver
112118
)
119+
if settings.security.enabled and agent_factory.settings.tools.cognite:
120+
if not agent_factory.settings.tools.cognite.client_secret:
121+
raise ValueError(
122+
"Cognite client secret must be provided in order to register the Cognite tools, "
123+
"when the security is enabled."
124+
)
125+
confidential_app = msal.ConfidentialClientApplication(
126+
settings.security.client_id,
127+
authority=settings.security.authority,
128+
client_credential=agent_factory.settings.tools.cognite.client_secret,
129+
)
130+
COGNITE_SCOPES = [f"{agent_factory.settings.tools.cognite.base_url}/.default", "offline_access"]
113131

114132
GraphDBHealthchecker(agent_factory.graphdb_client)
115133

@@ -399,36 +417,22 @@ async def get_auth_config(
399417
)
400418

401419

402-
if settings.security.enabled and agent_factory.settings.tools.cognite:
403-
if not agent_factory.settings.tools.cognite.client_secret:
404-
raise ValueError(
405-
"Cognite client secret must be provided in order to register the Cognite tools, "
406-
"when the security is enabled."
407-
)
408-
confidential_app = msal.ConfidentialClientApplication(
409-
settings.security.client_id,
410-
authority=settings.security.authority,
411-
client_credential=agent_factory.settings.tools.cognite.client_secret,
412-
)
413-
COGNITE_SCOPES = [f"{agent_factory.settings.tools.cognite.base_url}/.default", "offline_access"]
420+
def exchange_obo_for_cognite(user_access_token: str) -> dict:
421+
result = confidential_app.acquire_token_silent(COGNITE_SCOPES, account=None)
422+
if result:
423+
logging.debug("Cognite token acquired.")
424+
return result["access_token"]
414425

415-
416-
def exchange_obo_for_cognite(user_access_token: str) -> dict:
417-
result = confidential_app.acquire_token_silent(COGNITE_SCOPES, account=None)
418-
if result:
419-
logging.debug("Cognite token acquired.")
420-
return result["access_token"]
421-
422-
logging.debug("Acquiring token for Cognite using OBO.")
423-
result = confidential_app.acquire_token_on_behalf_of(
424-
user_assertion=user_access_token,
425-
scopes=COGNITE_SCOPES,
426-
)
427-
if "access_token" not in result:
428-
error_message = f"Failed to obtain OBO token for Cognite: {result}"
429-
logging.error(error_message)
430-
raise HTTPException(status_code=401, detail=error_message)
431-
return result
426+
logging.debug("Acquiring token for Cognite using OBO.")
427+
result = confidential_app.acquire_token_on_behalf_of(
428+
user_assertion=user_access_token,
429+
scopes=COGNITE_SCOPES,
430+
)
431+
if "access_token" not in result:
432+
error_message = f"Failed to obtain OBO token for Cognite: {result}"
433+
logging.error(error_message)
434+
raise HTTPException(status_code=401, detail=error_message)
435+
return result
432436

433437

434438
# noinspection PyUnusedLocal

‎src/talk2powersystemllm/app/trouble.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1345,7 +1345,7 @@ Sample Response Body:
13451345
},
13461346
"sample_sparql_queries": {
13471347
"enabled": true,
1348-
"sparql_query_template": "PREFIX retr: <http://www.ontotext.com/connectors/retrieval#>\nPREFIX retr-index: <http://www.ontotext.com/connectors/retrieval/instance#>\nPREFIX qa: <https://www.statnett.no/Talk2PowerSystem/qa#>\nSELECT ?question ?query {{\n [] a retr-index:{connector_name} ;\n retr:query \"{query}\" ;\n retr:limit {limit} ;\n retr:entities ?entity .\n ?entity retr:score ?score;\n qa:question ?question.\n ?template qa:paraphrase ?entity;\n qa:querySparql ?query.\n FILTER (?score > {score})\n}}\nORDER BY DESC(?score)\n",
1348+
"sparql_query_template": "PREFIX retr: <http://www.ontotext.com/connectors/retrieval#>\nPREFIX retr-index: <http://www.ontotext.com/connectors/retrieval/instance#>\nPREFIX qa: <https://www.statnett.no/Talk2PowerSystem/qa#>\nSELECT (REPLACE(GROUP_CONCAT(?q; separator=\"@\"), \"(.*?)@.*\", \"$1\") AS ?question) ?query {{\n SELECT ?q ?query ?score {{\n [] a retr-index:{connector_name} ;\n retr:query \"{query}\" ;\n retr:limit 100;\n retr:entities ?entity .\n ?entity retr:score ?score;\n qa:question ?q.\n ?template qa:paraphrase ?entity;\n qa:querySparql ?query.\n FILTER (?score > {score})\n }}\n ORDER BY DESC(?score)\n}}\nGROUP BY ?query\nLIMIT {limit}\n",
13491349
"connector_name": "qa_dataset"
13501350
},
13511351
"retrieve_data_points": {

‎src/talk2powersystemllm/qa_dataset/min_retrieval_limit.py‎

Lines changed: 0 additions & 48 deletions
This file was deleted.

0 commit comments

Comments
 (0)