Re: [cti-users] Re: EXTERNAL: [cti-users] Re: [STIX] STIX in JSON

I have to second this statement:  "that XPath is nearly impossible to work with in code".  We are using Java which has fantastic Xml support but combining JAXB and xpaths has been tedious.  

I think that having XPath within the data is one of the oddities of the model that binds it to an XML implementation.  Json has an equivalent of json path in some tools but using the concept of path within the model when everything else in the STIX model uses references.  I think the easiest implementation would be to just remove the Controlled_Structure element all together.  Let it apply to the node it belongs to (package, report, or indicator).

Kurt Zettel
Chief Architect
BrightPoint Security, formerly Vorstack

On Thu, Aug 20, 2015 at 4:14 PM, Jordan, Bret <> wrote:

From a JSON standpoint it is not really that difficult to make the conversion, especially on the STIX side of the house, I have done it and others have done it. JSON even supports schema validation for those interested in that (schema validation works today for JSON TAXII). CybOX will be a bit harder to do, but with the clean up and simplification that Ivan and Trey are working on, this should be possible as well.  Imagine how much easier CybOX would be to deal with in JSON... :)   

The only other problem I see is with the way data markings are currently done.  However, you could argue that this problem exists outside of JSON and that XPath is nearly impossible to work with in code.  To solve this problem I have proposed that we look at making data markings a top level object that can just be referenced by the elements that need it. This should solve the 95% use case. For the rest, those with very elaborate data markings, we will need to make sure they are taken care of. 

I believe that adopting JSON will lower the cost of entry for vendors, product managers, and developers to the point where we will get across the chasm and go mainstream with STIX. From what I heard at RSA and Blackhat as I walked the floor and talked with vendors (CEOs, CVOs, CTOs) is that, "if we only had JSON support we would be more willing to adopt STIX".  

I envision a day when there are hundreds or thousands of apps on the various APP stores that support cyber threat intelligence and at least 15-20 new startups all dealing with CTI and my "SoC of the future".  I think we can make it happen.  We just need to lower the cost of entry and give the product managers and developers what they want, which is clear when they tell me "can I have anything but XML".  

So I push for JSON.  

Thanks,

Bret

Bret Jordan CISSP
Director of Security Architecture and Standards | Office of the CTO

Blue Coat Systems

PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050

"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg." 

On Aug 20, 2015, at 14:34, Otto Fowler <> wrote:

Bret,

Can you speak to what you see as the current obstacles for supporting JSON as a complete binding?  Is it the security marking etc? 

From: <> on behalf of "Jordan, Bret"

Date: Thursday, August 20, 2015 at 3:30 PM

To: "", ""

Subject: EXTERNAL: [cti-users] Re: [STIX] STIX in JSON

Great question Chris, and very timely.

As many of you know, I have been beating the drum for JSON support for a long time, nearly 18 months now. I seem to get several private emails a week from the community asking about it. Even Facebook is using JSON for their ThreatExchange platform.
  

Where we stand today, to the best of my knowledge is:

1) STIX 1.2.1 will be ratified with OASIS using the current XML binding as the only reference implementation.  However, the STIX team as always maintained that you can do your own reference implementation.  And to that end several companies and
 groups have done just that.

2) There is support in the current Python APIs for a to_dict() to make it JSON.  I believe Soltra is using this on the back side as they store their STIX data in JSON format.  However, this to_dict() it is not a standardized form of JSON.  It
 was just put together by the developer of the API and some of the constructs are more XMLized JSON.

3) Several vendors and startups are doing their own STIX based JSON implementations for all of the reasons I have brought up in the past.  Intelworks is one of the few that have made public announcements about it.  The rest are still not public
 and thus I can not speak about them.  

4) JSON based TAXII is done and the specification can be found here [1].  I worked on this specification with lots of people and it has gone through several revisions in the TAXII community.  I even have fully working APIs for JSON based TAXII.

5) I am hoping that for STIX 2.0, we can have an official JSON binding for STIX that everyone can use.  But in the mean time, I would contact Intelworks and do what they are doing.  My project, FreeSTIX [2] is going to follow what Intelworks has
 done.

Everyone in this community that would like to see a JSON based STIX implementation should speak up.

[1] https://github.com/freetaxii/json-schema

[2] 
https://github.com/freestix 

Thanks,

Bret

Bret Jordan CISSP

Director of Security Architecture and Standards | Office of the CTO

Blue Coat Systems

PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050

"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg." 

On Aug 20, 2015, at 10:53, Christopher Roblee <> wrote:

Signed PGP part

I would like to resurrect this conversation on the new list.

Standard JSON bindings for STIX would greatly simplify interoperability

for us, and facilitate broader incorporation of the schema into our data

models. It sounds like this is the case for other vendors as well.

@Bret: could you please update the group as to where we are in terms of

getting JSON support in the standard libraries?

Thanks,

Chris

On 6/16/15 4:11 PM, Jordan, Bret wrote:

> I fully and 100% support your proposal as outlined below.

>

> Thanks,

>

> Bret

>

>

>

> *Bret Jordan CISSP*

> Director of Security Architecture and Standards | Office of the CTO

> Blue Coat Systems

> PGP Fingerprint: 62A6 5999 0F7D 0D61 4C66 D59C 2DB5 111D 63BC A303

> "Without cryptography vihv vivc ce xhrnrw, however, the only thing

that can not be unscrambled is an egg."

>

>> On Jun 16, 2015, at 16:58, Aharon Chernin <

<mailto:>> wrote:

>>

>> I do not see this discussion as XML vs JSON. In fact, we are one of

those companies who use primarily JSON then convert to XML right before

it goes out our pipes. This is a discussion around STIX interoperability.

>>

>>

>> Consumers chose the winners. This choice is not necessarily related

to ease of vendor implementation. Some good examples of this are VHS vs

Betamax or the success of the SCAP standards outside of the US Federal

government. If we give users a poor STIX experience, by making STIX

incompatible with STIX, then users are going to spread negativity about

STIX within the user community. They will instead chose products that

"just work", instead of products who are doing the right thing by

adopting STIX. Don't get me wrong, I am all for reducing vendor

complexity, but user experience comes first.

>>

>> Let me give you a poor user experience with STIX:

>> 1) ​OASIS ratifies new JSON binding of STIX for STIX 1.x

>> 2) Consumers now begin to discover that tools that have "adopted STIX

1.x" cannot communicate with each other

>> 3) OASIS releases STIX 2.0, which is not backwards compatible with

STIX 1.x. (I support this btw)

>> 4) Users now have STIX 1.x tools that cannot talk to each other and

STIX 2.x tools that cannot talk to STIX 1.x enabled products

>>

>> Aren't we supposed to be increasing interoperability?

>>

>> Discussion suggestions:

>> *) We discuss the future of STIX 1.x in OASIS. We can discuss holding

off new features in STIX 1.x - and only bug fix. We leave the only

agreed upon binding as XML to allow all STIX 1.x client/servers to

continue communications. At this point we begin to immediately begin

work on STIX 2.x.

>> *) We discuss using JSON as the default binding type for STIX 2.0

(which I would support btw). We already assume that STIX 1.x and 2.x

would be incompatible. This is our perfect chance to change exchange

formats.

>>

>> The above proposal only breaks interoperability between major STIX

versions. Major version incompatibility can be easily explained to users

and in some cases is even expected by users.

>>

>>

>> Aharon Chernin

>> CTO

>> *SOLTRA* | An FS-ISAC & DTCC Company

>> 18301 Bermuda green Dr

>> Tampa, fl 33647

>> 813.470.2173 | 

<mailto:><mailto:>

>> www.soltra.com <http://www.soltra.com/>

>> -------------------------

>> *From:* Jordan, Bret <

<mailto:>>

>> *Sent:* Tuesday, June 16, 2015 4:25 PM

>> *To:* Aharon Chernin

>> *Cc:* 

<mailto:>

>> *Subject:* Re: [STIX] Intelworks implementation of STIX in JSON

>>

>> The steps as I see them:

>>

>> 1: Finalize the JSON implementations for the various UML spec(s)

>> 2: Implement the JSON version in the various APIs, including the

MITRE Python/JAVA libraries (we also need libraries for a lot of other

languages to really go mainstream, not everyone uses or likes Python for

commercial products)

>>

>> At this point the market can decide which becomes the main method

people end up using.   My honest guess is it will be JSON, simple

because it will be easier and faster for web developers, app developers,

and open-source developers to use.  Most companies I have spoken with

today are already doing some sort of JSON based STIX on the back end and

then trying to do some conversion to XML to send it over the wire.  So

why not just send it over the wire in JSON format?  Seems like a way to

get people up and moving more quickly.

>>

>> When I look at it, if we sum up the total number of lines of code

that have been written so far for STIX, it will amount to less than 5%

of the sum total amount of code that has yet to be written.  One of the

major goals of the standard should be easy of implementation and use.

If the standard is too hard to implement, then people will not do it or

they will only do selective parts of it.

>>

>> The success of STIX and TAXII, in the end, will be based on the

number of platforms and projects that implement it.  If we get across

the chasm and go mainstream, then in my view we can expect to see

hundreds of apps on the various app stores and IMO, all of them will be

using JSON based STIX.

>>

>> Thanks,

>>

>> Bret

>>

>>

>>

>> *Bret Jordan CISSP*

>> Director of Security Architecture and Standards | Office of the CTO

>> Blue Coat Systems

>> PGP Fingerprint: 62A6 5999 0F7D 0D61 4C66 D59C 2DB5 111D 63BC A303

>> "Without cryptography vihv vivc ce xhrnrw, however, the only thing

that can not be unscrambled is an egg."

>>

>>> On Jun 16, 2015, at 13:55, Aharon Chernin <

<mailto:>> wrote:

>>>

>>> As it stands today, 99% of all STIX and TAXII traffic is XML based.

I am afraid we risk fragmenting the STIX community. My hope is that STIX

users (users: not vendors) don't have to experience a world where STIX

enabled products don't talk to each other due to one vendor picking JSON

over XML. Users won't blame the data exchange format, they will blame

STIX. We need to avoid that.

>>>

>>> One of the ways we can mitigate this is by including support for

JSON within the libraries used by most of the vendors who are already

creating STIX content. Mitre, are there plans to integrate the emerging

JSON format of STIX into PythonSTIX?

>>>

>>>

>>> Aharon Chernin

>>> CTO

>>> *SOLTRA* | An FS-ISAC & DTCC Company

>>> 18301 Bermuda green Dr

>>> Tampa, fl 33647

>>> 813.470.2173 | 

<mailto:><mailto:>

>>> www.soltra.com <http://www.soltra.com/>

>>> -------------------------

>>> *From:* Jordan, Bret <

<mailto:>>

>>> *Sent:* Tuesday, June 16, 2015 1:08 PM

>>> *To:* 

<mailto:>

>>> *Subject:* Re: [STIX] Intelworks implementation of STIX in JSON

>>>

>>> This is such great news.  I will be working very closely with

Interworks and others interested in JSON to get my implementation in

perfect alignment.

>>>

>>> I would challenge all of you interested in JSON, or that have

already started using JSON based STIX / TAXII in your implementations to

join us.  We need this community to push for a JSON based working group

in OASIS and we need all of your best ideas to help make this successful

as quickly as possible.

>>>

>>>

>>> Thanks,

>>>

>>> Bret

>>>

>>>

>>>

>>> *Bret Jordan CISSP*

>>> Director of Security Architecture and Standards | Office of the CTO

>>> Blue Coat Systems

>>> PGP Fingerprint: 62A6 5999 0F7D 0D61 4C66 D59C 2DB5 111D 63BC A303

>>> "Without cryptography vihv vivc ce xhrnrw, however, the only thing

that can not be unscrambled is an egg."

>>>

>>>> On Jun 16, 2015, at 08:09, Joep Gommers <

<mailto:>> wrote:

>>>>

>>>> Dear all,

>>>>

>>>> With the excellent work going on from @Bret Jordan on STIX in JSON,

we thought it helpful to share Intelworks approach to STIX in JSON and

ensure the community learned from our mistakes and investments. Props to

list- and team member @Wouter Bolsterlee for his work on this!

>>>>

>>>> *In short, lessons learned*

>>>>

>>>>   * Compound structures are objects

>>>>   * Attributes and child elements are key/value pairs

>>>>   * Relations are nested objects (or arrays of objects)

>>>>   * Flat is better than nested

>>>>   * And some ID and corner cases – see below

>>>>

>>>> Full details further down in this email. Your feedback is much

appreciated.

>>>>

>>>> We do have work-in-progress libraries available for (store-less)

bi-directional transformation of XML, JSON and YAML notations – which

might help those implementing STIX in JSON down the road. If you’d like

to know more, please contact me off-list.

>>>>

>>>> Best regards,

>>>> Joep

>>>>

>>>> Founder & CEO

>>>> Intelworks – Intelligence Powered Defence

>>>> www.intelworks.com <http://www.intelworks.com/>

>>>>

>>>> Find me at

>>>> +31 615489825

>>>> @joepgommers

>>>>

>>>>

>>>> ====

>>>>

>>>> The STIX language uses quite a few advanced XML modelling

techniques (multiple namespaces, xsi:type substitutions in instance

documents, QName identifiers, and so on), making it quite complex to

work with/implement. The JSON format used by Intelworks tries to be much

simpler to work with. Structurally it mirrors most of the original XML

tree structure, but the resulting tree structures are not identical

since the JSON representation favours flat objects over nested structures.

>>>>

>>>>

>>>>  
     Compound structures are objects

>>>>

>>>> In general, each compound structure is converted into a JSON object

(dict in Python). These objects always have a|type| key to indicate the

type of the structure:

>>>>

>>>> {

>>>>   "type": "indicator",

>>>>   "...": "..."

>>>> }

>>>>

>>>> Each of the main STIX constructs (see the STIX architecture

<http://stixproject.github.io/getting-started/whitepaper/#architecture>)

is represented as a JSON object. The |type| keys used are:

>>>>

>>>> Defining schema  
  XML Schema type  
  Object |type| field

>>>> STIX (Core)  
  |STIXType|  
  |package|

>>>> STIX (Campaign)  
  |CampaignType|  
  |campaign|

>>>> STIX (Course of Action)  
  |CourseOfActionType|  
  |course-of-action|

>>>> STIX (Exploit Target)  
  |ExploitTargetType|  
  |exploit-target|

>>>> STIX (Incident)  
  |IncidentType|  
  |incident|

>>>> STIX (Indicator)  
  |IndicatorType|  
  |indicator|

>>>> STIX (TTP)  
  |TTPType|  
  |ttp|

>>>> STIX (Threat Actor)  
  |ThreatActorType|  
  |threat-actor|

>>>> CybOX  
  |ObservableType|  
  |observable|

>>>>

>>>> Secondary constructs use these additional types (this list is NON

EXHAUSTIVE! And just a representation of potential)

>>>>

>>>> Defining schema  
  XML Schema type  
  Object |type| field

>>>> STIX (Common)  
  |IdentityType|  
  |identity|

>>>> STIX (Common)  
  |InformationSourceType|  
  |information-source|

>>>> STIX (Common)  
  |StatementType|  
  |statement|

>>>> STIX (Course of Action)  
  |ObjectiveType|  
  |objective|

>>>> STIX (Indicator)  
  |ValidTimeType|  
  |valid-time|

>>>> STIX (Markings)  
  |MarkingSpecificationType|

|marking-specification|

>>>> STIX (Markings)  
  |MarkingStructureType| (and extensions)

|marking-structure|

>>>> STIX (TTP)  
  |InfrastructureType|  
  |infrastructure|

>>>> STIX (TTP)  
  |MalwareInstanceType|  
  |malware-instance|

>>>> STIX (TTP)  
  |ResourceType|  
  |resource|

>>>> STIX (TTP)  
  |ToolInformationType|  
  |tool-information|

>>>> STIX (TTP)  
  |VictimTargetingType|  
  |victim-targeting|

>>>> CybOX  
  |MeasureSourceType|  
  |measure-source|

>>>> CybOX  
  |ObjectType|  
  |cybox-object|

>>>> CybOX  
  |ToolInformationType|  
  |tool-information|

>>>>

>>>>

>>>>  
     Attributes and child elements are key/value pairs

>>>>

>>>> Both the attributes and child elements defined for a compound

structure usually map to additional key/value pairs of the JSON objects:

>>>>

>>>> {

>>>>   "type": "indicator",

>>>>   "negate": false,

>>>>   "title": "This is the title."

>>>> }

>>>>

>>>>

>>>>  
     Relations are nested objects (or arrays of objects)

>>>>

>>>> For one-to-one relations, the value is a nested object, and the key

is a singular noun (|observable| in the example):

>>>>

>>>> {

>>>>   "type": "indicator",

>>>>   "observable": {

>>>>  
   "type": "observable",

>>>>  
   "...": "..."

>>>>   },

>>>>   "...": "..."

>>>> }

>>>>

>>>> For one-to-many relations, the value is a JSON array containing the

child objects, and the key is a plural noun (|indicators| in the example):

>>>>

>>>> {

>>>>   "type": "package",

>>>>   "indicators": [

>>>>  
   {

>>>>  
     "type": "indicator",

>>>>  
     "...": "..."

>>>>  
   },

>>>>  
   {

>>>>  
     "type": "indicator",

>>>>  
     "...": "..."

>>>>  
   }

>>>>   ],

>>>>   "...": "..."

>>>> }

>>>>

>>>> Additionally, the many |RelatedXYZ| constructs (and the surrounding

container objects) in STIX are also flattened: the target of the

relation is the child object (or a list of those), and any additional

relationship information is embedded into the child object(s):

>>>>

>>>> {

>>>>   "type": "indicator",

>>>>   "indicated_ttps": [

>>>>  
   {

>>>>  
     "type": "ttp",

>>>>  
     "relationship": "...",

>>>>  
     "relationship_information_source": "...",

>>>>  
     "...": "..."

>>>>  
   },

>>>>  
   {

>>>>  
     "type": "ttp",

>>>>  
     "relationship": "...",

>>>>  
     "relationship_information_source": "...",

>>>>  
     "...": "..."

>>>>  
   }

>>>>   ],

>>>>   "...": "..."

>>>> }

>>>>

>>>> See also the notes about nesting below.

>>>>

>>>>

>>>>  
     Flat is better than nested

>>>>

>>>> The STIX XML representation is deeply nested, partly due to the way

XML is typically used. The JSON representation tries to be a bit more

pragmatic and adheres to the "flat is better than nested" adage.

>>>>

>>>> In practice, this means that nested container structures are

flattened as much as possible. Unnecessary container structures are

simply removed. For example, the |<stix:Indicators>| container structure

used in the XML representation does not exist as such in the JSON

representation, since using an array is sufficient.

>>>>

>>>> To further reduce the number of nested objects, various XML

constructs using container elements with (optional) attributes are

flattened into the parent object by using multiple related keys. This is

best explained using an example.

>>>>

>>>> For example, the |StructuredTextType| used in both STIX and CybOX

is basically a string that can optionally carry a|structuring_format|

attribute. A naive conversion would require a nested object to represent

this:

>>>>

>>>> {

>>>>   "type": "...",

>>>>   "description": {

>>>>  
   "structuring_format": "html",

>>>>  
   "value": "Description goes here."

>>>>   },

>>>>   "...": "..."

>>>> }

>>>>

>>>> Since the |structuring_format| is optional, this approach would

often result in a small nested object with only a single key/value pair

(the |value|). To avoid this, /objectivistix/ takes an alternative

approach using two related keys in the containing object:

>>>>

>>>> {

>>>>   "type": "...",

>>>>   "description": "Description goes here.",

>>>>   "description_structuring_format": "html",

>>>>   "...": "..."

>>>> }

>>>>

>>>> In case the |structuring_format| is not specified, the

|description_structuring_format| key/value pair would simply not be present:

>>>>

>>>> {

>>>>   "type": "...",

>>>>   "description": "Description goes here.",

>>>>   "...": "..."

>>>> }

>>>>

>>>>

>>>>  
     ID handling

>>>>

>>>> All |id| and |idref| attributes in STIX XML are not simply string

values, but qualified names (QName in XML), meaning that they contain a

namespace prefix which resolves to a namespace URI. To avoid any

explicit mappings for these prefixes and their associated namespace URI,

the JSON representation always expresses |id| and |idref| values in

their canonical form using the so-called Clark notation

<http://www.jclark.com/xml/xmlns.htm>, which looks like this:

|{http://example.com/ns/uri}local-name|.

>>>>

>>>> The top level object may optionally contain an |id_namespaces|

mapping that maps prefixes to namespace URIs. This mapping will be used

to determine the prefixes used for |id| and |idref| attribute values

when converting the object to XML, as illustrated by the example below:

>>>>

>>>> {

>>>>   "type": "package",

>>>>   "id":

"{http://example.org/}Package-b3ba766b-d3e6-4d92-82b2-5940f0cb763c",

>>>>   "id_namespaces": {

>>>>  
   "example": "http://example.org/"

>>>>   }

>>>> }

>>>> <stix:STIX_Package

>>>>   xmlns:stix="http://stix.mitre.org/stix-1"

>>>>   xmlns:example="http://example.com/"

>>>>   id="example:Package-b3ba766b-d3e6-4d92-82b2-5940f0cb763c">

>>>>   …

>>>> </stix:STIX_Package>

>>>>

>>>> In case no |id_namespaces| mapping is present, a unique namespace

prefix will be used instead. The |id_namespaces| can safely be left out

with no semantical loss, since the prefix is arbitrary and only used for

serialized XML data, and not for the in-memory model.

>>>>

>>>>

>>>>  
     Special conversion notes

>>>>

>>>>  *

>>>>

>>>>  
   STIX package header

>>>>

>>>>  
   The package header is not treated as a first-class structure.

Since the |STIX_Header| construct only applies to|STIX_Package|, it is

merged completely into the main |package| object (this avoids having an

additional nested object for the header):

>>>>

>>>>  
   {

>>>>  
     "type": "package",

>>>>  
     "description": "Description goes here.",

>>>>  
     "...": "..."

>>>>  
   }

>>>>  *

>>>>

>>>>  
   Structured text

>>>>

>>>>  
   The |StructuredTextType| construct is not transformed into a

child object. Instead, the keys |foo| and

(optionally)|foo_structuring_format| are added to the containing object.

>>>>

>>>>  *

>>>>

>>>>  
   Observable composition

>>>>

>>>>  
   An ‘observable composition’ structure does not result in a

nested object for the composition itself. Instead, the|composition| key

contains the child objects, and the |composition_operator| specifies the

operator:

>>>>

>>>>  
   {

>>>>  
     "type": "indicator",

>>>>  
     "observable": {

>>>>  
    
   "composition_operator": "or",

>>>>  
    
   "composition": [

>>>>  
    
     {

>>>>  
    
    
   "type": "observable",

>>>>  
    
    
   "...": "..."

>>>>  
    
     },

>>>>  
    
     {

>>>>  
    
    
   "type": "observable",

>>>>  
    
    
   "...": "..."

>>>>  
    
     }

>>>>  
    
   ]

>>>>  
     },

>>>>  
     "...": "..."

>>>>  
   }

>

--

Chris Roblee

Director of Engineering

TruSTAR Technology

Mobile: +1 781 248 2828

OpenPGP key ID: 2C9D0D20