I have no issue with the concept of getting
an MVP, out so long as the future is considered so we don't box ourselves
into a corner. I have already given very specific future-facing examples
of problems this MVP introduces, that need to be thought through. We have
already been down this path, with several breaking changes between TAXII
2.0 and TAXII 2.1, and parties coming to the fore saying they are "waiting"
for 2.1 so they don't have to re-do implementations.
If TAXII 2.2 introduces more breaking
changes because we did not consider all the use cases and tried to rush
things out, then we are going to rapidly start to loose faith in the marketplace.
I'll also again re-iterate that it is
very much debatable even if this is the actual MVP we need at this point
in time. It depends highly on where in the intelligence ecosystem you sit.
I will take a FireEye product as an example - if I have a file hash and
during a hunt I want to query Helix using it's API for all of the sightings
of that file hash, this MVP does not help me. We have no way to do this
extremely basic use case in STIX today. I think this global query use case
is *more* critical than being able to de-reference a relationship only
inside a single collection... collections after all are often mirrored
wholesale.
RE query - this is the exact use case
that the STIX Extension process is meant to solve, by forcing folks down
a path where proposals are implemented and tested and vetted in the wild
before they make it into the standard. I think that anything we do along
the lines of Query should have to be proven in the market before it makes
it into the spec along with the potential for future breakages.
- Jason Keirstead Lead Architect - IBM Security Connect
www.ibm.com/security
"Things may come to those who wait, but only the things left by those
who hustle." - Unknown
From:
Gary Jay Katz <>
To:
Bret Jordan <>,
Jason Keirstead <>
Cc:
""
<>
Date:
11/19/2018 07:14 PM
Subject:
Re: [cti-taxii]
Re: [EXT] Re: [cti-taxii] TAXII Query / Search
Sent by:
<>
Jason,
That s a good question.
One which I can provide my response, but I believe requires a larger
discussion, which is part of the reason why I believe we should get an
MVP out now, rather than waiting. To Bret s point, I d rather not
wait 6-12 months to get a release out.
Currently we have ways
to easily search for a STIX object. It does not allow for STIX querying
or the complex use cases we have identified, but it does allow someone
to easily retrieve an SDO. And that is a good thing. There
should be an easy way to perform a search, whether that is for an object
or a relationship and it should be easy for a consumer to use that functionality
to traverse the graph and easy for a producer to implement such functionality.
Not every producer will have a backend database (or the requirements)
to allow for complex querying. I truthfully do not see the issue
with allowing restful methods to easily traverse the graph and then later
add in full query support where you could accomplish the same thing in
addition to much more powerful queries. People may disagree, and
we can and should debate this as TAXII query is designed, but I personally
do not believe that the full TAXII query support needs to follow the same
pattern as is defined for retrieving an SDO or SRO.
Just my opinion, but hope it helps explain
where my thoughts are.
-Gary
From: Bret Jordan <>
Date: Monday, November 19, 2018 at 4:41 PM
To: Jason Keirstead <>, Gary Katz <>
Cc: "" <>
Subject: Re: [cti-taxii] Re: [EXT] Re: [cti-taxii] TAXII Query / Search
This provides a solution that we can have now. At the current
rate any more advanced solution is at least 6-12 more months away. Yes,
we may need to do things differently in the future, but that is okay. We
can always deprecate features as needed. But I figured a more RESTful design
would get us up and going sooner rather than later.
By my count the following individuals support the proposal
I have submitted:
Myself
Drew
Nicholas
Sean
Gary
The individuals that do not support it:
Jason
I am curious to hear what others have to say. Can everyone
please chime in on this debate.
Bret
From: Jason Keirstead <> Sent: Monday, November 19, 2018 2:25:37 PM
To: Gary Jay Katz
Cc: Bret Jordan;
Subject: Re: [cti-taxii] Re: [EXT] Re: [cti-taxii] TAXII Query / Search
Gary - how would you propose in
the future we can extend or expand this approach, without redoing it from
scratch? This is the problem with having query-specific endpoints or query parameters
such as "related to" - they can't be extended or added to in
any logical way.
It will result in either no way to actually do combinations of parameters,
or an unworkable mishmash of normative text quantifying all of the edge
cases of which endpoints can be combined with which others to get what
behaviour. Sent from IBM Verse
Gary Jay Katz --- Re: [cti-taxii]
Re: [EXT] Re: [cti-taxii] TAXII Query / Search ---
From:
"Gary
Jay Katz" <>
To:
"Jason
Keirstead" <>, "Bret Jordan"
<>
Cc:
Date:
Mon,
Nov 19, 2018 4:55 PM
Subject:
Re:
[cti-taxii] Re: [EXT] Re: [cti-taxii] TAXII Query / Search
Jason,
I believe you are correct,
there are a number of complex use cases that this would not meet. For
those that remember back to some of the early face-to-face meetings, I
pushed for trying to find a solution that met all of these solutions in
the first release. I now believe that this needs to be taken in parts.
There may be certain TAXII implementations that only need to and
are able to support basic search and pivoting, while others will be built
to perform complex querying such as what you are suggesting.
Having an ability to
easily query relationships and pivot off of them which does not support
the more complex use cases, does not mean that the capability needs to
be rewritten or that those complex use cases are invalid. We just
need ways to do the simple things simply, while eventually put in place
the ability to do the complex capabilities.
Currently we have no ability
to query relationships or pivot. This is a blocker for anyone trying
to provide the baseline capabilities to support the STIX model using TAXII.
I would suggest we get the baseline capabilities in place to allow
releasing a minimal viable product in the near-term while continuing to
work towards more advanced query capabilities in a future release.
-Gary
From: <>
on behalf of Jason Keirstead <>
Date: Monday, November 19, 2018 at 3:38 PM
To: Bret Jordan <>
Cc: "" <>
Subject: [cti-taxii] Re: [EXT] Re: [cti-taxii] TAXII Query / Search
The problem with this approach of "getting
something done in a a few days", is it is highly likely it will have
to be re-done in the future because it isn't considering the other use
cases.
For example - In the future how can I find all sightings related to Object
Foo that match the indicator 1.2.3.4, with your proposal? Seeing how any
future "match the indicator" portion would be an entirely different
endpoint than the "related-to" endpoint, I now can't combine
them - and therefore would have to make multiple - perhaps dozens, hundreds,
or even thousands of round-trips to the server to collect this information.
- Jason Keirstead Lead Architect - IBM Security Connect
www.ibm.com/security
"Things may come to those who wait, but only the things left by those
who hustle." - Unknown
From: Bret
Jordan <>
To: Jason Keirstead
<>
Cc: ""
<>
Date: 11/19/2018
04:27 PM
Subject: Re:
[EXT] Re: [cti-taxii] TAXII Query / Search
Jason, the proposal you have submitted is valid, and one potential solution.
However, it represents a fair bit of complexity on the specification
and on implementations. There are also a lot of unanswered questions about
how it would actually be done. To support this, we would also need to create
several conformance levels for it, which is non-trivial.
Per your own statements, over the past 12 months I have seen very little
to no public support for it. As such, I am trying to find a
simpler
solution
that we can do now and then look at the more complicated
solution later. I fear if we try to do a more complicated solution
now, then TAXII 2.1 will still be many months away from being done.
The proposal I have worked on with Drew and Gary is something we can do
very easily in a matter of days, which could enable us to finish TAXII
2.1 within the next month.
But all of this is up to the TC to decide. We can see what kind of
feedback we get, and see where the majority of TC members sit. But at this
stage, I would personally (not as a chair or editor) prefer a more simple
solution that can be used now.
Bret
From: Jason Keirstead <> Sent: Monday, November 19, 2018 1:06:11 PM
To: Bret Jordan
Cc:
Subject: [EXT] Re: [cti-taxii] TAXII Query / Search
Hi Brett - Terry and I have an alternative proposal we've been asking the
TC to have a look at here
https://docs.google.com/document/d/1Cy_9Bh5tKEkDHGg2iv5c3AwriqVr7ygbKXWOv4-uHxs/edit#heading=h.1jcqb6vc5y7z
As we lay out in the document, there are a number of issues TAXII has with
query, and I think the proposal you have below is insufficient to solve
the use cases we have in Github: We first started working on this last Christmas - so it's been almost an
entire year. We've been struggling to get feedback on the document and
would like the TC to comment on it. It is in my opinion a very simple proposal
- yet it allows query to be extensible for many future use cases.
We need to avoid endpoint pollution as one tries to bolt-on new endpoints
and arguments for every kind of query parameter under the sun without any
plan for the future use cases we have to meet.
Issue
15 - As a User, I want to traverse the STIX graph over TAXII in an efficient
manner...
Issue
7 - Need ability to request related objects in one request to a distance
of 1(?)
Issue
5 - Add support to get embedded objects in a single request, dereference...
Issue
6 - Add ability to find all objects related to a particular STIX object
ID
Issue
4 - TAXII Observed Data Query
Also, I think "the biggest element holding people back" is very
much open to interpretation and what persona you fill. From the perspective
of most any SOAR-based security tool integrating with a TAXII server, the
biggest element holding me back is ability to find indicators that match
an observation, or find observation that match an indicator.
Finding out who is related to a STIX object is very much a TIP-centric
use case (although I will add that that use case is also addressed in our
above proposal as well). My point is that we should not be trying to come out with a lot of piecemeal-endpoints
without thinking of the bigger picture of query. - Jason Keirstead Lead Architect - IBM Security Connect
www.ibm.com/security
"Things may come to those who wait, but only the things left by those
who hustle." - Unknown
From: Bret
Jordan <>
To: ""
<>
Date: 11/19/2018
03:26 PM
Subject: [cti-taxii]
TAXII Query / Search
Sent by: <>
All, Over the past year I have heard from several people that they are unable
to implement TAXII because there is no way to search for relationships.
Yes, there are lots of other things that people would like to do with a
query request, but the ability to search for a relationship seems to be
the single biggest missing element holding people back from implementing
TAXII. I have talked with Drew and Gary about how we could solve this in the very
short term and would like to propose the following solution to the TC.
If the TC agrees with this, Drew and I can add this to Working Draft 05
in the next week and submit Working Draft 05 to the TC for Review and CSD
ballot the first of December.
Proposal 1) We create an endpoint called: <api-root>/collections/<id>/relationships/related-to/<stix-id>/
This endpoint would return any SROs where the source_ref or target_ref
matched the supplied STIX ID. This would be a very simple database query,
and would not require a lot of computation work. This endpoint would
be mandatory to implement for all TAXII Servers as defined in the conformance
section.
2) We add an optional URL parameter for the relationships endpoint called:
?deref=true This optional URL parameter would tell the server to automatically send
the objects that are referenced in the SROs that are being returned. From
a conformance standpoint, this URL parameter would be optional to implement.
If a client makes a request with that URL parameter and the server has
not implemented it, the server would respond with an HTTP 501 Not Implemented
error code and a TAXII error message that says the URL parameter is not
implemented. From a database performance standpoint, this URL parameter would require
that the server perform multiple database queries for each request and
would require the server to do some book keeping to ensure that it does
not send the same object multiple times. However, this feature would eliminate
the overhead of parsing multiple RESTful requests from the client as it
comes back and asks for each of the objects one at a time.
Conclusion We have some support for doing this already as this would be easy to implement
in the specification and in code, and would solve a major blocker that
has been identified. I would be curious to know if anyone else would support
this or be against this. This does not mean that we will not look
at a more elaborate pattern based / property based query solution in the
future.
Bret
This email and any attachments thereto
may contain private, confidential, and/or privileged material for the sole
use of the intended recipient. Any review, copying, or distribution of
this email (or any attachments thereto) by others is strictly prohibited.
If you are not the intended recipient, please contact the sender immediately
and permanently delete the original and any copies of this email and any
attachments thereto.
This email and any attachments thereto may contain private,
confidential, and/or privileged material for the sole use of the intended
recipient. Any review, copying, or distribution of this email (or any attachments
thereto) by others is strictly prohibited. If you are not the intended
recipient, please contact the sender immediately and permanently delete
the original and any copies of this email and any attachments thereto.
