encoding/xml: XML namespace prefixes not fully supported #6800

gopherbot · 2013-11-20T10:09:55Z

by anthonyeden:

It is possible to create a prefix on elements by simply defining the prefix as part of
the name in the XMLName declaration, but the xmlns attribute is output without the
prefix. Example: http://play.golang.org/p/KeIHB4_z5L

Thus it would seem that the encoding/xml package can only deal with document-level
namespaces, not nested namespaces, is this correct?

rsc · 2013-11-27T18:44:28Z

Comment 1:

Labels changed: added go1.3maybe.

rsc · 2013-12-04T01:31:05Z

Comment 2:

Labels changed: added release-none, removed go1.3maybe.

rsc · 2013-12-04T01:43:14Z

Comment 3:

Labels changed: added repo-main.

anacrolix · 2013-12-10T14:31:16Z

Comment 4:

This generates:
<example:foo xmlns="urn:example"></example:foo>
Am I correct in believing you require:
<example:foo xmlns:example="urn"></example:foo>
?

gopherbot · 2013-12-10T15:58:42Z

Comment 5 by anthonyeden:

Yes, you are correct.

gopherbot · 2013-12-10T16:00:14Z

Comment 6 by anthonyeden:

FWIW, I'm creating an RFC5730-RFC5733 implementation in Go, which is what triggered this
requirement.

anacrolix · 2014-01-04T16:16:57Z

Comment 7:

Downstream issue
https://bitbucket.org/anacrolix/dms/issue/19/samsung-doesnt-like-response-to

anacrolix · 2014-01-04T16:18:11Z

Comment 8:

And https://bitbucket.org/anacrolix/dms/issue/25/get-soap-responses-to-work-with-upnp.
Perhaps this will help explain the shortcomings here.

gopherbot · 2014-11-08T23:56:40Z

Comment 9 by camilo.aguilar:

The lack of good support for namespace prefixes does seem to be affecting us here too
hooklift/gowsdl#17

jordan2175 · 2015-01-06T22:32:20Z

As I commented on #9519 can we get the ability for xml.Marshal to use prefixes?

rogpeppe · 2015-02-25T18:03:28Z

Fixed by https://go-review.googlesource.com/2660

jordan2175 · 2015-02-28T06:37:10Z

Can we verify that it will in fact work with an XML Namespace aspect like (note the various XMLNS prefixes “stix:, indicator:, cybox:”):

<stix:Indicators>
    <stix:Indicator id="example:Indicator-d81f86b9-975b-bc0b-775e-810c5ad45a4f" xsi:type="indicator:IndicatorType">
        <indicator:Title>Malicious site hosting downloader</indicator:Title>
        <indicator:Type xsi:type="stixVocabs:IndicatorTypeVocab-1.0">URL Watchlist</indicator:Type>
        <indicator:Observable id="example:Observable-ee59c28e-d922-480e-9b7b-a79502696505">
            <cybox:Object id="example:URI-b13ae3fc-80af-49c2-9de9-f713abc070ba">
                <cybox:Properties xsi:type="URIObj:URIObjectType" type="URL">
                    <URIObj:Value condition="Equals">http://x4z9arb.cn/4712</URIObj:Value>
                </cybox:Properties>
            </cybox:Object>
        </indicator:Observable>
    </stix:Indicator>
</stix:Indicators>

/stix:STIX_Package

Thanks,
Bret
PGP Fingerprint: 62A6 5999 0F7D 0D61 4C66 D59C 2DB5 111D 63BC A303
"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."

On Feb 25, 2015, at 11:03 AM, Roger Peppe <notifications@github.com mailto:notifications@github.com> wrote:

Fixed by https://go-review.googlesource.com/2660 https://go-review.googlesource.com/2660
—
Reply to this email directly or view it on GitHub #6800 (comment).

rogpeppe · 2015-03-01T08:58:17Z

I'm sorry, I don't quite understand what you mean when you ask whether "it will in fact work with an XML Namespace aspect like ...".

jordan2175 · 2015-03-02T03:33:17Z

Given the XML example, can your patch parse it correctly and then re-marshall it back in to the same thing? I just want to make sure that it works with all of the namespace prefixes. Thanks SO MUCH for the patch and for fixing this issue.

Thanks,
Bret
PGP Fingerprint: 62A6 5999 0F7D 0D61 4C66 D59C 2DB5 111D 63BC A303
"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."

On Mar 1, 2015, at 1:58 AM, Roger Peppe notifications@github.com wrote:

I'm sorry, I don't quite understand what you mean when you ask whether "it will in fact work with an XML Namespace aspect like ...".

—
Reply to this email directly or view it on GitHub #6800 (comment).

davecheney · 2015-03-02T03:44:40Z

There are no xmlns attributes in the fragment you presented. That looks
like xml schema information.

On Mon, Mar 2, 2015 at 2:33 PM, jordan2175 notifications@github.com wrote:

Given the XML example, can your patch parse it correctly and then
re-marshall it back in to the same thing? I just want to make sure that it
works with all of the namespace prefixes. Thanks SO MUCH for the patch and
for fixing this issue.

Thanks,
Bret
PGP Fingerprint: 62A6 5999 0F7D 0D61 4C66 D59C 2DB5 111D 63BC A303
"Without cryptography vihv vivc ce xhrnrw, however, the only thing that
can not be unscrambled is an egg."

On Mar 1, 2015, at 1:58 AM, Roger Peppe notifications@github.com
wrote:

I'm sorry, I don't quite understand what you mean when you ask whether
"it will in fact work with an XML Namespace aspect like ...".

—
Reply to this email directly or view it on GitHub <
https://github.com/golang/go/issues/6800#issuecomment-76588584>.

—
Reply to this email directly or view it on GitHub
#6800 (comment).

jordan2175 · 2015-03-02T03:58:39Z

Here is an example with them.

<stix:STIX_Package
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:stix="http://stix.mitre.org/stix-1"
xmlns:indicator="http://stix.mitre.org/Indicator-2"
xmlns:cybox="http://cybox.mitre.org/cybox-2"
xmlns:AddressObject="http://cybox.mitre.org/objects#AddressObject-2"
xmlns:cyboxVocabs="http://cybox.mitre.org/default_vocabularies-2"
xmlns:stixVocabs="http://stix.mitre.org/default_vocabularies-1"
xmlns:ExampleNamespace="http://example.com/"
xsi:schemaLocation="
http://stix.mitre.org/stix-1 http://stix.mitre.org/XMLSchema/core/1.0.1/stix_core.xsd
http://stix.mitre.org/Indicator-2 http://stix.mitre.org/XMLSchema/indicator/2.0.1/indicator.xsd
http://cybox.mitre.org/default_vocabularies-2 http://cybox.mitre.org/XMLSchema/default_vocabularies/2.0.1/cybox_default_vocabularies.xsd
http://stix.mitre.org/default_vocabularies-1 http://stix.mitre.org/XMLSchema/default_vocabularies/1.0.1/stix_default_vocabularies.xsd
http://cybox.mitre.org/objects#AddressObject-2 http://cybox.mitre.org/XMLSchema/objects/Address/2.0.1/Address_Object.xsd"
id="ExampleNamespace:STIXPackage-33fe3b22-0201-47cf-85d0-97c02164528d"
version="1.0.1">
<stix:STIX_Header>
    <stix:Title>Example watchlist that contains IP information.</stix:Title>
    <stix:Package_Intent xsi:type="stixVocabs:PackageIntentVocab-1.0">Indicators - Watchlist</stix:Package_Intent>
</stix:STIX_Header>
<stix:Indicators>
    <stix:Indicator xsi:type="indicator:IndicatorType" id="ExampleNamespace:Indicator-33fe3b22-0201-47cf-85d0-97c02164528d">
        <indicator:Type xsi:type="stixVocabs:IndicatorTypeVocab-1.0">IP Watchlist</indicator:Type>
        <indicator:Description>Sample IP Address Indicator for this watchlist. This contains one indicator with a set of three IP addresses in the watchlist.</indicator:Description>
        <indicator:Observable  id="ExampleNamespace:Observable-1c798262-a4cd-434d-a958-884d6980c459">
            <cybox:Object id="ExampleNamespace:Object-1980ce43-8e03-490b-863a-ea404d12242e">
                <cybox:Properties xsi:type="AddressObject:AddressObjectType">
                    <AddressObject:Address_Value condition="Equals" apply_condition="ANY">10.0.0.0##comma##10.0.0.1##comma##10.0.0.2</AddressObject:Address_Value>
                </cybox:Properties>
            </cybox:Object>
        </indicator:Observable>
    </stix:Indicator>
</stix:Indicators>
</stix:STIX_Package>

davecheney · 2015-03-02T04:04:00Z

I'm sorry, github ate your upload. Can you put it in a file an attach or
use a pastebin service ?

On Mon, Mar 2, 2015 at 2:58 PM, jordan2175 notifications@github.com wrote:

Silly of me... Here is an example with them.

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:stix="http://stix.mitre.org/stix-1"
xmlns:indicator="http://stix.mitre.org/Indicator-2"
xmlns:cybox="http://cybox.mitre.org/cybox-2"
xmlns:AddressObject="http://cybox.mitre.org/objects#AddressObject-2"
xmlns:cyboxVocabs="http://cybox.mitre.org/default_vocabularies-2"
xmlns:stixVocabs="http://stix.mitre.org/default_vocabularies-1"
xmlns:ExampleNamespace="http://example.com/"
xsi:schemaLocation="
http://stix.mitre.org/stix-1
http://stix.mitre.org/XMLSchema/core/1.0.1/stix_core.xsd
http://stix.mitre.org/Indicator-2
http://stix.mitre.org/XMLSchema/indicator/2.0.1/indicator.xsd
http://cybox.mitre.org/default_vocabularies-2
http://cybox.mitre.org/XMLSchema/default_vocabularies/2.0.1/cybox_default_vocabularies.xsd
http://stix.mitre.org/default_vocabularies-1
http://stix.mitre.org/XMLSchema/default_vocabularies/1.0.1/stix_default_vocabularies.xsd
http://cybox.mitre.org/objects#AddressObject-2
http://cybox.mitre.org/XMLSchema/objects/Address/2.0.1/Address_Object.xsd"
id="ExampleNamespace:STIXPackage-33fe3b22-0201-47cf-85d0-97c02164528d"
version="1.0.1">
stix:STIX_Header
stix:TitleExample watchlist that contains IP information./stix:Title
Indicators - Watchlist/stix:Package_Intent
/stix:STIX_Header
stix:Indicators

IP Watchlist/indicator:Type
indicator:DescriptionSample IP Address Indicator for this watchlist. This
contains one indicator with a set of three IP addresses in the watchlist.
/indicator:Description

10.0.0.0##comma##10 #10
#10 #10
/AddressObject:Address_Value
/cybox:Properties
/cybox:Object
/indicator:Observable
/stix:Indicator
/stix:Indicators
/stix:STIX_Package

—
Reply to this email directly or view it on GitHub
#6800 (comment).

jordan2175 · 2015-03-02T23:23:52Z

I see that. I edited the post above to fix it. I also copied it to pastebin,here
http://pastebin.com/kbtDUWdE

jordan2175 · 2015-03-03T17:20:47Z

Here is the example from their website as well.

http://stix.mitre.org/about/example_watchlist.html http://stix.mitre.org/about/example_watchlist.html

Thanks,
Bret
PGP Fingerprint: 62A6 5999 0F7D 0D61 4C66 D59C 2DB5 111D 63BC A303
"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."

On Mar 1, 2015, at 9:04 PM, Dave Cheney notifications@github.com wrote:

I'm sorry, github ate your upload. Can you put it in a file an attach or
use a pastebin service ?

On Mon, Mar 2, 2015 at 2:58 PM, jordan2175 notifications@github.com wrote:

Silly of me... Here is an example with them.

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:stix="http://stix.mitre.org/stix-1"
xmlns:indicator="http://stix.mitre.org/Indicator-2"
xmlns:cybox="http://cybox.mitre.org/cybox-2"
xmlns:AddressObject="http://cybox.mitre.org/objects#AddressObject-2"
xmlns:cyboxVocabs="http://cybox.mitre.org/default_vocabularies-2"
xmlns:stixVocabs="http://stix.mitre.org/default_vocabularies-1"
xmlns:ExampleNamespace="http://example.com/"
xsi:schemaLocation="
http://stix.mitre.org/stix-1
http://stix.mitre.org/XMLSchema/core/1.0.1/stix_core.xsd
http://stix.mitre.org/Indicator-2
http://stix.mitre.org/XMLSchema/indicator/2.0.1/indicator.xsd
http://cybox.mitre.org/default_vocabularies-2
http://cybox.mitre.org/XMLSchema/default_vocabularies/2.0.1/cybox_default_vocabularies.xsd
http://stix.mitre.org/default_vocabularies-1
http://stix.mitre.org/XMLSchema/default_vocabularies/1.0.1/stix_default_vocabularies.xsd
http://cybox.mitre.org/objects#AddressObject-2
http://cybox.mitre.org/XMLSchema/objects/Address/2.0.1/Address_Object.xsd"
id="ExampleNamespace:STIXPackage-33fe3b22-0201-47cf-85d0-97c02164528d"
version="1.0.1">
stix:STIX_Header
stix:TitleExample watchlist that contains IP information./stix:Title
Indicators - Watchlist/stix:Package_Intent
/stix:STIX_Header
stix:Indicators

IP Watchlist/indicator:Type
indicator:DescriptionSample IP Address Indicator for this watchlist. This
contains one indicator with a set of three IP addresses in the watchlist.
/indicator:Description

10.0.0.0##comma##10 #10
#10 #10
/AddressObject:Address_Value
/cybox:Properties
/cybox:Object
/indicator:Observable
/stix:Indicator
/stix:Indicators
/stix:STIX_Package

—
Reply to this email directly or view it on GitHub
#6800 (comment).

—
Reply to this email directly or view it on GitHub #6800 (comment).

c4milo · 2015-03-18T01:02:47Z

@rogpeppe, did this fix land in Go 1.4.x?

c4milo · 2015-03-18T01:40:49Z

@rogpeppe, I'm having problems marshaling, when attempting to consume a webservice that is expecting namespaces declared at the top along with prefixes, and each inner element using the prefix as opposed to the actual namespace attribute. You can find more details about this issue here: hooklift/gowsdl#17

notzippy · 2015-03-18T04:10:27Z

@c4milo This fix is not in go 1.4.1

jordan2175 · 2015-03-18T06:19:35Z

Here is another example that the current XML bindings can not process.. It is a bit different than the current issues, I think, or maybe it is related. If I take out all but one of the xmlns attributes, then it works.

<taxii_11:Discovery_Response 
    xmlns:taxii="http://taxii.mitre.org/messages/taxii_xml_binding-1"
    xmlns:taxii_11="http://taxii.mitre.org/messages/taxii_xml_binding-1.1"
    xmlns:tdq="http://taxii.mitre.org/query/taxii_default_query-1" 
    message_id="32898" 
    in_response_to="1">
</taxii_11:Discovery_Response>

doomedramen · 2015-03-18T16:35:34Z

I am trying to parse (small snippet):

<w:document xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main"></w:document>

but I get: error: expected element type <w:document> but have

I could change

XMLName xml.Name `xml:"w:document"`

to

XMLName xml.Name `xml:"document"`

but that breaks everything else.

c4milo · 2015-03-18T16:40:39Z

@wookoouk this fix is going to land in Go 1.5. So, we are going to still see this failing in Go 1.4.x

doomedramen · 2015-03-18T16:41:11Z

Is it available in HEAD?

c4milo · 2015-03-18T16:43:51Z

@wookoouk yes, master branch, aka tip, should have it.

doomedramen · 2015-03-18T16:44:22Z

Cheers @c4milo

notzippy · 2015-03-18T16:46:57Z

I found that the master did not make much difference, I used another package https://github.com/jteeuwen/go-pkg-xmlx to read the xml, correct the name space, and output the xml

jordan2175 · 2015-03-18T16:48:27Z

To everyone that is writing the code / patches and everyone that is contributing examples of things that are not working, THANKS!!!! What a great community. Lets make Golang the best!

doomedramen · 2015-03-18T16:58:42Z

yep, sadly the same for me:

go version devel +6ffed30 Wed Mar 18 15:14:37 2015 +0000 darwin/amd64
[xmd] go run xmd.go tomd test.docx                                                                       16:58:32  ☁  master ☂ ✖ ⚡ ✭
word/document.xml
error: expected element type <w:document> but have <document>%

c4milo · 2015-03-18T17:19:17Z

This issue should be re-opened then.

mikioh · 2015-03-19T00:27:34Z

Please open a new issue if necessary. The above additional information confuses the issue whether the applied fix generates a new issue (breaks existing code and we should revert the fix) or just not fixing undiscovered, already reported issues.

If you are saying that the applied change 2660 should be reverted for some reason, please write on this issue so.
If you are saying that the applied change 2660 does not fix your issue, please search existing issues and write additional information for contributors, or open a new issue if necessary.

Running issue filters with "encoding/xml in:title" still reports there're 19 open issues.

jordan2175 · 2015-03-19T01:06:22Z

I do not believe this issue has actually been fixed with patch 2660. Especially when you consider the fact that issues #9519 and #9775 were closed and folded in to this issue and neither one of them has been addressed. That means until they are also fixed, this issue is not done and should be reopened.

mikioh · 2015-03-19T02:26:51Z

@jordan2175,

Fine, re-open both #9519 and #9775 instead. But please add more useful information for contributors in #9519 and #9775. As @rogpeppe said in above:

I'm sorry, I don't quite understand what you mean when you ask whether "it will in fact work with an XML Namespace aspect like ...".

gopherbot added new labels Nov 8, 2014

bradfitz removed the new label Dec 18, 2014

mikioh mentioned this issue Jan 6, 2015

encoding/xml: support for XML namespace prefixes #9519

Open

jordan2175 mentioned this issue Jan 8, 2015

encoding/xml: missing nested namespace not handled #7113

Closed

c4milo mentioned this issue Jan 17, 2015

NRCS Envelope Fault hooklift/gowsdl#17

Open

c4milo mentioned this issue Feb 5, 2015

encoding/xml: Unmarshal does not properly handle NCName in XML namespaces #9775

Open

rogpeppe closed this as completed Feb 25, 2015

mikioh added this to the Go1.5 milestone Feb 25, 2015

mikioh removed priority-triage labels Feb 25, 2015

olivere mentioned this issue Jul 1, 2015

encoding/xml: Serializing XML with namespace prefix #11496

Open

rakyll mentioned this issue Sep 16, 2015

encoding/xml: brittle support for matching a namespace by identifier or url #12624

Open

golang locked and limited conversation to collaborators Jun 25, 2016

gopherbot added the FrozenDueToAge label Jun 25, 2016

encoding/xml: XML namespace prefixes not fully supported #6800

encoding/xml: XML namespace prefixes not fully supported #6800

Comments

gopherbot commented Nov 20, 2013

rsc commented Nov 27, 2013

rsc commented Dec 4, 2013

rsc commented Dec 4, 2013

anacrolix commented Dec 10, 2013

gopherbot commented Dec 10, 2013

gopherbot commented Dec 10, 2013

anacrolix commented Jan 4, 2014

anacrolix commented Jan 4, 2014

gopherbot commented Nov 8, 2014

jordan2175 commented Jan 6, 2015

rogpeppe commented Feb 25, 2015

jordan2175 commented Feb 28, 2015

rogpeppe commented Mar 1, 2015

jordan2175 commented Mar 2, 2015

davecheney commented Mar 2, 2015

jordan2175 commented Mar 2, 2015

davecheney commented Mar 2, 2015

jordan2175 commented Mar 2, 2015

jordan2175 commented Mar 3, 2015

c4milo commented Mar 18, 2015

c4milo commented Mar 18, 2015

notzippy commented Mar 18, 2015

jordan2175 commented Mar 18, 2015

doomedramen commented Mar 18, 2015

c4milo commented Mar 18, 2015

doomedramen commented Mar 18, 2015

c4milo commented Mar 18, 2015

doomedramen commented Mar 18, 2015

notzippy commented Mar 18, 2015

jordan2175 commented Mar 18, 2015

doomedramen commented Mar 18, 2015

c4milo commented Mar 18, 2015

mikioh commented Mar 19, 2015

jordan2175 commented Mar 19, 2015

mikioh commented Mar 19, 2015